Class 5: Checking an agent’s evidence
Tuesday, November 10, 2026
Have your class 4 homework (question, prediction, two commands with counts, and the passage that you read), your CSV, and your author’s folder class4-<author> at hand. If you have not yet done so, submit the class 4 evidence record and the CSV at the submission page (campus network only). The page lists the questions and predictions from the submitted records under “Submitted questions”.
If your author is not Akutagawa, also save and extract the Akutagawa folder class4-akutagawa. Section 4 uses it to rerun the recorded commands.
| What you use | From class 4 | Today |
|---|---|---|
| Text | Your author’s prepared corpus | The same corpus |
| Search | Your two commands, or the commands behind the saved outputs that you used | The commands that the agent proposes, and your reruns |
| Record | Question, prediction, counts, CSV rows 1–12 | The record of checked claims, and the report for the evidence assignment |
Plan the 90 minutes as follows. Before section 1, copy the headings in the exit note into your notes; each section fills in the items shown in the table. Section 3 is optional; if there is no time for it in class, try it afterwards.
| Section | Time | Exit note items |
|---|---|---|
| 1. Instructions | 10 minutes | Instructions |
| 2. Recordings | 30 minutes | A/B comparison, command record |
| 4. Reproducing a claim | 25 minutes | Checked claim, checked quotation |
| 5. Your own investigation and the exit note | 25 minutes | Unconfirmed explanation |
What did the class 4 labels show?
Nine worksheets were submitted by November 9. Eight have the same direction and movement labels as the sample worksheet for rows 2–7; the ninth differs only on row 4. These counts show agreement with the sample worksheet and do not establish that the judgments were made independently.
| Row | Expression | Submitted labels (worksheets) | Pairwise agreement | Method differing from the majority label |
|---|---|---|---|---|
| 2 | どこまでものぼって行けば |
Not stated whether it happens (Uu): 9 |
100% | Qwen: Happens / happened (CT+) |
| 3 | たぐりのぼり始めました |
Happens / happened (CT+): 9 |
100% | None |
| 4 | 上へ上へ一心によじのぼって来る |
Happens / happened (CT+): 8; Not stated whether it happens (Uu): 1 |
78% | None |
| 5 | 下りろ |
Not stated whether it happens (Uu): 9 |
100% | Qwen: Does not happen / did not happen (CT-) |
| 6 | まっさかさまに落ちてしまいました |
Happens / happened (CT+): 9 |
100% | None |
| 7 | 元の地獄へ落ちてしまった |
Happens / happened (CT+): 9 |
100% | None |
All nine worksheets have the same direction label for each passage. Among rules, Qwen, and Perplexity Decider, Qwen differs from the majority movement label on rows 2 and 5. The sample worksheet’s criterion assigns “Not stated whether it happens” (Uu) to movement expressed only as a command or condition.
The table below shows alternative readings that change the event being classified, the source of the judgment, or the scope of an expression of uncertainty.
| Row | Sample judgment | Alternative reading | Changed criterion | Wording to read |
|---|---|---|---|---|
| 2 | Not stated whether it happens (Uu) |
Happens / happened (CT+) |
Label the climbing that the story goes on to narrate, not the conditional clause | The next paragraph: 「こう思いましたから犍陀多は、…たぐりのぼり始めました」 (“thinking so, Kandata … began to climb”) |
| 4 | Happens / happened (CT+), judgment source: narrator |
Happens / happened (CT+), judgment source: Kandata |
Record whose perception of the event is described | 「ふと気がつきますと」 (“when he suddenly noticed”). The same 「ではございませんか」 in row 2’s paragraph reports the thread coming down as fact |
| 5 | Not stated whether it happens (Uu) |
Does not happen / did not happen (CT-) |
Label whether the ordered descent happens by the end of the story | The thread breaks right after the shout (row 6), and no one climbs down |
| 7 | Happens / happened (CT+) |
Probably happened (record this interpretation in your notes) | Apply the narrator’s conjecture 「でございましょう」 (“probably”) to the whole sentence | 「…落ちてしまったのが、御釈迦様の御目から見ると、浅間しく思召されたのでございましょう」. The fall itself was already narrated in row 6 |
When classifying row 4 under an alternative reading, explain how you read 「ではございませんか」 and record the criterion beside the label, as in the worksheet’s 「修正した基準」 (revised criterion) field.
“Probably happened” describes uncertainty in the narrator’s account. “Defer judgment” records that you cannot assign a worksheet label, with the reason written in the evidence field.
1. Turn your question into an instruction
An agent is a language model that uses tools, such as a search tool, to run commands in the working folder and explain their results. You check those explanations against the command outputs and the source text.
Write two instructions from one question. Use your class 4 question, or choose one from “Submitted questions” on the submission page if you do not have one. If the page is unavailable, use the Akutagawa example below.
For instruction A, write only the question, without naming files, a search tool, an analysis method, or the evidence that you want with the answer. For example:
In Akutagawa Ryūnosuke's fiction, how many times does 落ちる combine with 来る, and how many times with 行く? Which combination is more frequent?
When you try instruction A in section 3, use a fresh conversation without supplying the course folder or files. Save the first answer before adding any follow-up instruction.
For instruction B, name the tool, the data and where to obtain it, and the evidence that you want with the answer. An instruction in this form needs no other file or explanation, so it gives the same task in any folder on a computer where ling is installed. Use this form. Replace class4-akutagawa (in the folder name and in the URL), akutagawa.lingcorpus, the number of works, and the question with your own:
Investigate this question with the search tool ling, which is installed. Run ling --help for its syntax.
Data: use the prepared corpus akutagawa.lingcorpus, the works list works.csv, and the pattern file class4.lx in the folder class4-akutagawa. If the folder is missing, download and extract https://nlp.lang.osaka-u.ac.jp/dh/b/materials/class4-akutagawa.zip. Do not create or change any file other than that ZIP file and folder.
Question: In the 70 works in akutagawa.lingcorpus, how many times does 落ちる combine with 来る, and how many times with 行く? Which combination is more frequent?
Conditions: show every command that you run. For each command, show the output lines that your answer uses and, if you filtered the output, the command that selected them. Give the unit with every value (token count, work count, or rate per 10,000 verbs). List every match in a table with the work's title, the whole sentence, whether a person or thing changes position, and the reason for that judgment. For comparison, give the same counts for all verbs and the number of works that contain the matches.
| Part of the instruction | Role |
|---|---|
| Tool | Name the search tool and where its syntax is documented |
| Data | Specify the folder and files, where to obtain them, and that no other file may be created or changed |
| Question | One question that asks for counts |
| Conditions | Ask for the commands, output, and units, as in your own records from class 3 onward; also ask for a judgment of every match and for counts to compare with |
2. Read the recorded investigation
In this section you read four conversations: instruction A and instruction B, each entered with two agents (Claude Code and Codex). This page calls an answer to instruction A “response A” and the recorded conversation for instruction B “recording B”. All four are in Japanese.
Every conversation was started fresh on November 10, 2026, in an empty folder without the course files, and received the Japanese instruction from section 1 unchanged. The agents could run commands without asking for confirmation, and ling was available only in the instruction B conversations.
| Recording detail | Claude Code | Codex |
|---|---|---|
| Agent version | Claude Code 2.1.321 | Codex CLI 0.167.0 |
| Model | claude-opus-5-5 |
gpt-6.1-sol |
| Search tool (instruction B only) | ling 0.7.0 |
ling 0.7.0 |
Read response A
Response A answers instruction A, which names no files and no tool. Before opening it, write in your notes whether you expect the agent to give counts and, if so, where you expect it to get them.
| Response A | Last message | Whole conversation |
|---|---|---|
| Claude Code | Text | Recording |
| Codex | Text | Recording |
The two agents responded differently. Claude Code looked in the empty folder, gave no count because it found no texts, and asked which material to count and how. Codex searched the web and quoted 10 and 1 from this course’s class 4 page. Neither conversation contains a command that searches the 70-work corpus used in class.
Codex’s 10 and 1 equal the counts in the course corpus, but they are quoted from a page that already states the answer and are not the result of an investigation. The last message alone does not show this difference; check the searches and commands in the recording of the whole conversation.
The counts 10 and 1 make it look as if 落ちる tends to combine with 来る. Compare that idea with a second author. In authors.csv, the columns 落ちる_来る and 落ちる_行く give 10 and 1 for Akutagawa, and 0 and 3 for Dazai Osamu. With Dazai’s counts in view, consider whether you can say, for any author, that 落ちる tends to combine with 来る.
Watch recording B and record its commands
Before playing recording B, open Akutagawa’s saved output 来る/行く passages and find the 11 lines with 落ちる. Write in your notes how many of them you expect to describe an actual fall of a person or thing.
Recording B for Claude Code and recording B for Codex follow below. A recording shows the commands that the agent ran at one time in one box; this page calls one box a command block. Choose one recording B and, while you play it, write the agent’s commands in your notes with the form below.
The table shows how much to write.
| What to record | For which commands |
|---|---|
| The Command and Execution lines | Every command block in which the agent runs ling on the corpus (12 in the Claude Code recording, 3 in the Codex recording), and any ling command that the agent proposed but did not run (Execution: not run) |
| The other five lines | At least two commands, including the command for the claim that you check in section 4 |
| Files read and commands run | One list for the whole conversation, compared with what the agent states that it read, ran, and created or changed |
Copy this form into your notes, once for each command block.
Command: (the line as written)
Execution: run / not run / cannot determine
Count: (the output count, "not run", or "not applicable")
Unit stated by the agent: (the unit, or "not stated")
Unit the command counted: (the unit, "not run", or "not applicable")
Claim: (the agent's one-sentence claim based on this output, or "none")
Quotation: (the work's title and the passage, or "none")
If a block contains several ling lines, write all of them on the Command line. File reads, help pages, and searches of other files are not command records; put them in the list of files that the agent read. Use “cannot determine” for details that you cannot establish from the conversation.
To read the other parts of a block, such as head, jq, for, or 2>&1, use Read an agent’s commands. Its table of commands that create, change, or delete files is the list to check against the agent’s statement.
How do I fill in a command record?
This example records the block in Claude Code’s recording B that counts all verbs and the matches for 落ちる.
Command: rm /tmp/sum.tsv /tmp/sum.err; cd /work/class4-akutagawa && ling summarize --corpus akutagawa.lingcorpus --patterns class4.lx --top all --tsv | grep -E '^pos.*"VERB"|^deixis.*"(落ちる|)"' | grep -E 'VERB|落ちる'
Execution: run
Count: VERB 30878; 落ちる + 来る 10; 落ちる + 行く 1
Unit stated by the agent: トークン数 (tokens)
Unit the command counted: matches of the class4.lx patterns (pos is one token; deixis is a sequence of three tokens)
Claim: 落ちて来る is 10 tokens and 落ちて行く is 1 token (「10トークン(6作品、動詞1万語あたり3.24)」, 「1トークン(1作品、動詞1万語あたり0.32)」)
Quotation: none
On the Command line, copy the line in the box as written. This line deletes two files made in the previous block with rm, then narrows the output of ling summarize to three lines with grep. On the Count line, write the numbers at the right end of the three output lines.
This example lists the files read and the commands run in the same conversation.
- Files read:
README.txt,class4.lx, andworks.csv(its first five lines, its line count, and seven titles) - Help shown:
ling --help, and the help forsummarize,corpus, andmatches - Blocks that run
lingon the corpus: 12, fromling corpus inspectto the count of works - Files created:
class4-akutagawa.zipand the extracted folder;/tmp/sum.tsvand/tmp/sum.err, deleted in the block above - Compared with the agent’s statement: it states that it created the two
/tmpfiles against the condition and deleted them, which matches the commands. It does not mentionwhich jq, which checked thatjqwas installed
In these commands, summarize counts pattern matches grouped by the specified key, not works.
Use Play to show the conversation in order, or the arrows to move between blocks. You can also open each recording in a new window: Claude Code’s recording B, Codex’s recording B.
The two B recordings give the same counts, but for one of the 11 matches they disagree on whether a person or thing changes position. Compare the tables in the two final answers and find that match. Read the passage and decide which judgment you agree with.
After the recordings, compare response A with recording B.
| What to compare | Response A: Claude Code | Response A: Codex | Recording B: both |
|---|---|---|---|
| Material identified | None; the agent only looked in the empty folder | The counts for 70 works stated on the class 4 page | The 70 works in akutagawa.lingcorpus |
| Numerical result | No count given | 10 and 1, quoted from the page | 10 matches for 落ちて来る, 1 for 落ちて行く |
| How it was counted | Not counted; the agent asked which material and which spellings to count | The agent did not count | The lemma pattern of class4.lx |
| What another reader can repeat | Nothing; the material and the way of counting must be chosen first | Open the quoted page | Rerun the commands and check the passages in the assigned text |
How do the October 27 recordings differ?
The October 27 recordings (Japanese, English) were made on the instructor’s computer, where the agent could also read the source files of this course’s pages. Before proposing commands, the agent (Claude Code, Claude Fable 5.1, ling 0.5.0) read the class 4 page source: in the English recording, the sentence that gives the counts 10 and 1 for 落ちる. The instruction also differed from section 1 and included a condition to ask for approval before each command.
The conversation therefore does not show whether the numbers in the answer rest only on command output. The November 10 recordings were made in a folder from which the page sources could not be read, with the section 1 wording unchanged.
3. Investigate with your own agent as well (optional)
If a coding model is available on your own computer, try instructions A and B in two fresh conversations. Use the same model for both when possible.
Set the agent to ask for approval before running a command. The instruction not to create or change other files does not by itself stop a command from running; only the agent’s permission setting does.
- Record the agent or tool name, the model name if shown, and the date for each conversation.
- For A, start a fresh conversation in an empty folder without attaching course files. Enter only instruction A and save the first answer. Record any tools that the agent ran without being asked.
- For B, start a separate conversation in your working folder
dh-b-2026, which containsclass4-<author>, and enter instruction B. Before approving a command, check that it does not change or create files; ask for screen output instead of file writes. - Use the section 2 comparison table to record what evidence each answer provides. For B’s commands, also fill in the command record from section 2. If A admits uncertainty or supplies checkable evidence on its own, record that too.
- Save both conversations as .txt or .md files. Leave out API keys and personal conversations.
To check what the conditions in instruction B change, you can also start a third conversation in dh-b-2026 and enter instruction B without its Conditions paragraph. Approve only commands that do not change or create files, and record which commands, units, and quotations the agent supplies without those conditions.
The next question contains a premise that has not been checked: that 上る combines with 来る more often than with 行く. To see whether an agent checks that premise, enter the question in a fresh conversation without course files, and save the first answer.
Why does 上る combine with 来る more often than with 行く in Akutagawa Ryūnosuke's fiction?
For the 70 works, authors.csv gives 10 matches with 来る and 13 with 行く, so the premise does not hold in this corpus. Record whether the answer tries to check the premise or explains a reason without checking it.
If you do not use a model, use the comparison of response A and recording B from section 2, and use recording B for the checks in sections 4 and 5.
Which agent can I use?
You can use any agent that runs in a terminal and can execute commands in the working folder. With a model hosted by an external service, the instructions that you enter and the command outputs are sent to that service. For this exercise, enter only the text and the distributed files.
4. Reproduce one claim
From the final answer of the recording B that you chose in section 2, or of your own conversation B from section 3, choose one claim that includes a count, other than the claims checked in the 「疑惑」 example at the end of this section, the exit-note example, and the example report. Run the commands in the folder that contains the corpus named in the instruction. Replace a folder path in the recorded commands, such as /work, with the path of your own folder, and write the replacement in your notes. Keep the pattern, filters, and counting unit unchanged.
To rerun the commands from recording B, use the Akutagawa folder class4-akutagawa.
Without a terminal, inspect the class 4 saved outputs and the quoted source text. Record the inability to rerun the command as the reason for an unresolved result, and record the quotation check separately.
- Run the command behind the claim in your own terminal, and compare the counts.
- Display the quoted passage from the text with
--where '@document.title = TITLE'and check that it matches the quotation. - Record the result as one of the three outcomes in the table below.
| Outcome | What to record |
|---|---|
| Agreement | The same count and unit, and a quotation that matches the text |
| Disagreement | The count from your rerun or the wording you found in the text, and the condition that produced the difference (command, lemma set, unit, or range of works); if the cause is unknown, say so |
| Unresolved | Why you could not complete the check (missing command, a different file, unavailable source, and so on) |
A matching count does not establish the claim. As with あのぼんやり in class 3, a matched expression may fall outside the scope of your question. Read the quoted paragraph to decide.
Choose one more number in the agent’s response that no command output shows directly, such as a count that the agent obtained by reading passages or by adding rows. For example, the “91%” in Claude Code’s recording B is a number that the agent calculated from 10 and 11. Check it against the command outputs and the paragraphs opened from the reader links in works.csv, and record the result as one of the same three outcomes.
For example, these commands display passages from 「疑惑」, which both B recordings quote.
ling matches --corpus akutagawa.lingcorpus --where '@document.title = 疑惑' --pattern '[lemma=落ちる] [lemma=て] [lemma in {来る, 行く}].' --context 4 --color never
ling matches --corpus akutagawa.lingcorpus --where '@document.title = 疑惑' --pattern '[text ~ 梁].' --context 3 --color neverThe first line shows the passages in Akutagawa’s 「疑惑」 (Giwaku, “Doubt”) where 落ちる combines with 来る or 行く, with four tokens of context on each side. The distributed corpus gives 3 matches, all 落ちて来る. If the quotation does not match the pattern, search for a word from the quotation with [text ~ word], as in the second line.
Read the paragraph with the first match and the paragraph with the second and third matches in Soranoha.
5. Compare with your own investigation
Put the two commands from your class 4 homework beside the agent’s commands.
- If your question is the same as the agent’s, compare the counts and their conditions: the pattern, the lemma set, the
--whererange, and the unit. Record agreement or the difference that you found. If the question differs, choose one check that the agent made and that your homework did not, such as the count over all verbs for comparison or the count per work. Run that check for your question on your author’s corpus, and record the command and the count; without a terminal, use your author’s saved outputs and record that you did not rerun the commands. - Write down one passage that the agent found and that you missed, or one that you read and that the agent did not quote. If you found neither, record that and name the passages that you compared.
- Write out one sentence of the agent’s explanation that you could not confirm from the command outputs and the text. If you found none, record that and state which claims you checked.
Exit note
Record these six items in your notes (.txt or .md) under the headings below, and submit the notes at the submission page (campus network only). If you saved a conversation in section 3, submit that file as well.
| What to save | Contents |
|---|---|
| Instructions | Your instructions A and B from section 1; for the conversations that you checked, the instructions actually used and the agent, model, and date (supplied examples are identified in section 2) |
| A/B comparison | One claim from response A, the evidence it provides, and what recording B establishes or leaves unresolved; if you ran your own conversations, the same for them |
| Command record | The section 2 record for the commands in the recording B that you chose, or in your own conversation B: two lines for every command and all seven for at least two; the files read and commands run, compared with the agent’s statement |
| Checked claim | The section 4 claim, its command, the count from your rerun, and the outcome (agreement, disagreement, or unresolved); the number not shown in any command output and its outcome |
| Checked quotation | The work’s title and the result of checking against the text |
| Unconfirmed explanation | The sentence from section 5 and why it could not be confirmed, or that none was found among the claims that you checked |
Instructions (section 1):
A/B comparison (section 2):
Command record (section 2):
Checked claim (section 4):
Checked quotation (section 4):
Unconfirmed explanation (section 5):
What does a completed exit note look like?
This example entry uses Claude Code’s response A and recording B, Codex’s recording B, and Akutagawa’s corpus. It shows one reading; record your own checks and judgments.
| What to save | Example entry |
|---|---|
| Instructions | A and B: the section 1 forms with class4-akutagawa, 70 works, and the same question. Checked conversations: Claude Code’s response A and recording B, which received the Japanese wording of section 1 unchanged; Claude Code 2.1.321, claude-opus-5-5, November 10, 2026 |
| A/B comparison | Response A (Claude Code): no count is claimed. It states that the working folder /work is empty (「作業ディレクトリ /work は空です」) and asks which material and which forms to count. Evidence: two commands that list the folder’s contents. Recording B counts 10 and 1 tokens in the 70 works of akutagawa.lingcorpus by lemma. Akutagawa’s works favor 来る, but not every author’s do: authors.csv gives 0 and 3 for Dazai Osamu. Which of the 11 matches describe a fall is left to reading the passages |
| Command record | Claude Code’s recording B: the Command and Execution lines for all 12 blocks; all seven lines for the block in the section 2 example and for the block that counts works (how to read it); the list of files read and the comparison as in that example |
| Checked claim | Claim: 落ちて来る is 10 tokens in 6 works, and 落ちて行く 1 token in 1 work. Rerun in my class4-akutagawa folder with the same patterns, listing the distinct file names in column 3 instead of using jq: ling matches --corpus akutagawa.lingcorpus --tsv --limit all --pattern '[lemma=落ちる] [lemma=て] [lemma=来る].' gives 10 matches in 6 files, and [lemma=行く] 1 match in Akutagawa_R_Sutego.md. Outcome: agreement, with works counted as files. Number not shown in any output: 「11件のうち実際に下へ移動しているのは6件」 (“6 of the 11 actually move down”). In the 11 sentences, paper (「アグニの神」, 2), eaves and two beams (「疑惑」, 3), and a sailor’s corpse (「誘惑」) fall; the beams fall in a painting that the narrator describes. Outcome: agreement, with the two beams counted as depicted falls |
| Checked quotation | 「誘惑」, sentence 42: 「急にどこか空中から水夫の死骸が一つ落ちて来る。」 (“Suddenly a sailor’s corpse falls from somewhere in the air”). ling matches --corpus akutagawa.lingcorpus --where '@document.title = 誘惑' --pattern '[lemma=落ちる] [lemma=て] [lemma in {来る, 行く}].' --context 12 lists it in Akutagawa_R_Yuuwaku.md, and the paragraph has the same sentence. Outcome: agreement |
| Unconfirmed explanation | None found among the claims that I checked. Checked explanation: in Claude Code’s recording B, row 2 explains 「アグニの神」 sentence 125 「おや、紙切れが落ちて来たが、――もしや御嬢さんの手紙じゃないか?」 as 「1と同じ紙切れが、上から話し手の近くへ移動しています」 (“the same piece of paper as in row 1 moves from above to near the speaker”). The command output shows only the sentence, but the next paragraph states that 遠藤, who said these words, picks the paper up, so the explanation is confirmed |
Homework: assemble the classes 1–5 investigation into a report
Write the report under the six items of the evidence assignment. Submit it at the submission page (campus network only) before class 6 on Tuesday, November 17.
Use your earlier records (question, prediction, commands, and CSV passages) and today’s checks to write the six items, including your conclusion and the next change to your method. In item 5, choose a claim other than those checked in the section 4 examples, and write only the interactions and checks that you actually carried out. If your report includes a worksheet row that has an alternative reading in the class 4 results, state the criterion that you applied.