Class 4: Comparing authors and models

1. Set up your author’s prepared corpus

If you have not yet done so, submit your class 2 worksheet dh-b-movement.json at the submission page. Keep the page open: sections 5 and 6 use it.

If you no longer have dh-b-movement.json, save the sample worksheet and load it with 「保存したワークシートを開く」 (open a saved worksheet) in the worksheet of class 2 section 2. Rewrite the evidence of all seven rows in your own words, change the labels of any row you judge differently, then save and submit the file. Do not submit a file whose evidence is still the sample’s.

Save your author’s ZIP from the table below into the class 2 working folder dh-b-2026. Extracting the ZIP creates the author’s folder, such as class4-akutagawa, containing akutagawa.lingcorpus and the other files. On a computer without ling, first follow Set up ling.

Start your terminal in the folder that contains dh-b-2026. Run the following commands; if you have already extracted the ZIP with your file manager, skip the unzip line.

cd dh-b-2026
unzip class4-akutagawa.zip
cd class4-akutagawa
ling corpus inspect akutagawa.lingcorpus

Replace class4-akutagawa.zip and class4-akutagawa with your author’s ZIP and folder names from the table below. Replace akutagawa.lingcorpus with the prepared corpus filename inside that folder. If inspect reports the number of works in the table and finishes with complete:, continue to the next section.

The course uses a fixed selection of 380 works from Soranoha, grouped by 13 writers. The selection includes Kunikida’s translation of 「糸くず」 and Mori’s translations of 「冬の王」 and 「みれん」; works.csv records the original author and each writer’s role. The texts and bibliographic information come from the Soranoha release recorded in README.txt. Akutagawa’s selection contains 70 works, compared with 103 in class 3.

The first_published field preserves Soranoha’s first-publication information when available; year is extracted from that field and is blank when no year is recorded. Check the work’s Soranoha page before comparing publication periods.

2. Count the vertical-movement lemmas in your author

class4.lx combines the vertical and deixis definitions from class 3 with pos, which counts tokens by part of speech. Each [...] is one token, and brackets written in a row match consecutive tokens. The key: line above a definition sets what summarize counts.

(?<name> ...) is a named capture: it gives the matched part a name that you can use later. In deixis, two parts have names.

key: [$verb.lemma, $deictic.lemma] groups counts by the combination of these two lemmas.

key: [lemma]
deixis_compound := [upos=VERB & lemma ~ /(来る|行く)$/ & lemma not in {来る, 行く, 出来る, 心行く}].

authors.csv was built from class4.lx so that all 13 authors are compared with the same set and the same denominator. Leave class4.lx as distributed. To run the set you revised in class 3, copy it to class4-mine.lx, edit the copy, and write class4-mine.lx beside every count taken from it.

ling check --patterns class4.lx
ling summarize --corpus akutagawa.lingcorpus --patterns class4.lx --top all > summary.txt
grep -w pos summary.txt | grep VERB
grep -w vertical summary.txt
grep -w vertical summary.txt | awk '{s+=$1} END {print s}'

The awk command sums and prints the counts from the first column, one per vertical lemma.

A vertical count is the number of tokens whose lemma is in the set, not the number of movements. In Akutagawa’s 「文章」 (Bunshō, “writing”; Akutagawa_R_Bunshou.md), 肉の落ちた (“thin”) matches 落ちる but describes a face, with no movement. The next command lists the first 40 落ちる hits with the file name, the sentence number within the work, and the surrounding text:

ling matches --corpus akutagawa.lingcorpus --tsv --context 8 --limit 40 --pattern '[lemma=落ちる].' | cut -f3,5,9-

Authors differ in the length of their works, so compare counts as a rate per 10,000 verbs. The rate is the vertical total divided by the VERB count, times 10,000. In Akutagawa’s 70 works, the vertical total is 406, the VERB count is 30,878, and the rate is 131.5.

1. Calculate the vertical total, the VERB count, and the rate, and record them in your notes.
2. Write down the top three vertical lemmas.
3. Find one 落ちる hit in your author in which nothing moves, and write down its wording and file name.
4. Compare the three values with your author’s row in authors.csv, columns verbs, vertical, and vertical_per_10k_verbs. If a value differs, write down which command gave the different count.

3. Check how often verbs combine with 来る and 行く

grep -w deixis summary.txt | grep -E '上る|落ちる|下りる'

In Akutagawa’s 70 works, 落ちる combines with 来る 10 times and 行く once, and 上る combines with 来る 10 times and 行く 13 times. A pair absent from the output has a count of 0.

1. Compare your author’s four counts with the authors.csv columns in the section 2 table.
2. Does 落ちる also combine more frequently with 来る in your author? Write one sentence.

4. Choose a work and read two passages

ling matches --corpus akutagawa.lingcorpus --tsv --pattern '[lemma in {上る, 落ちる, 下りる}] [lemma=て] [lemma in {来る, 行く}].' | cut -f3 | sort | uniq -c | sort -rn | head
ling matches --corpus akutagawa.lingcorpus --where '@document.title = 杜子春' --tsv --context 12 --pattern '[lemma in {上る, 落ちる, 下りる}] [lemma=て] [lemma in {来る, 行く}].' | cut -f3,5,9-

The first command counts, per file, the passages in which 上る, 落ちる, or 下りる is followed by て来る or て行く: three consecutive tokens, as in Figure 3.

The second shows the passages of one work: the file name, the sentence number within the work, and the surrounding text. Replace akutagawa.lingcorpus with your corpus filename and 杜子春 with the title value of the chosen file in works.csv.

「杜子春」 (Toshishun) has four such passages. In Kunikida’s corpus, only 「少年の悲哀」 (Kodomo no kanashimi, “childhood sorrow”) (Kunikida_D_Kodomonokanashi.md) has two. When no work by your author has at least two such passages, run the second command without --where '@document.title = 杜子春' and choose passages from two works. Without a terminal, the saved 来る/行く passages file for your author lists the same output.

1. Choose the file with the most passages, select two passages, and read the full paragraph containing each one: open the work’s reader link in works.csv and search for the wording shown by ling. The linked examples above open the relevant paragraph directly.
2. Classify them with the same three judgments as in class 2 and add them as CSV rows 11–12. Write the work’s title in part and the wording shown by ling in expression. The other column values are those in the class 3 homework table.
3. Write in your notes the file name and sentence number of each passage, who moves, and where the narrator watches from.

5. Compare the class’s results

authors.csv holds the section 2 and 3 values for all 13 authors.

The submission page shows the label distributions, deferrals, and pairwise agreement for rows 2–7, separately for direction and the movement judgment. Outlines mark the passages where your labels differ from the majority.

1. Write down the authors with the highest and lowest rates per 10,000 verbs, and one possible reason based on the kind of work or the period.

2. Write down any author whose works do not combine 落ちる with 来る more often than with 行く.

3. Choose the row with the lowest agreement. Look in the submitted criteria list for a criterion that would resolve the disagreement for that row. If none does, write your own in the worksheet’s 「修正した基準」 (revised criterion) field.

   When you save and resubmit the file, the list updates within a minute. If the page is unavailable, or no submissions are listed yet, compare your labels in worksheet section 4 with a partner’s saved file or with the sample worksheet, and write a criterion for a row where the judgments differ in your notes. A comparison with the sample is agreement with one reading, not the class’s agreement rate.

The rate counts lemma hits, including hits like the one you noted in section 2, and depends on the dictionary’s decisions and on the choice of the vertical set. The selection includes children’s fiction and three translations, and first-publication information is incomplete. State these conditions whenever you describe a difference between authors.

6. Compare a model’s extractions with your labels

For the seven worksheet passages, compare your classifications with the results from rules, the LLM Qwen3.5-4B, and Perplexity Decider. For extraction from whole paragraphs, compare the results from rules, GLiNER2.5, and Qwen3.5-4B. The notebook displays saved results, so no model download is required. The following command installs only marimo. Its interface is in Japanese.

Unpack the extraction comparison notebook (ZIP) and keep all files together, including notebook-theme.css and the hidden .marimo.toml. Run this command from that folder and open the URL it prints in your browser. The notebook opens in a browser window. Keep the submission page open in another tab or window and switch between them in the steps below.

uvx --python 3.14 marimo==0.25.0 edit --no-sandbox movement-extraction.py

If uvx is not recognized, first install uv by following Run a course notebook.

1. On the submission page, choose one passage with a wavy underline in the 規則 (rules), Qwen, or Decider column. The underline marks a method label that differs from the label most readers chose. If the page is unavailable or no passage is underlined, choose row 2, どこまでものぼって行けば, or row 5, 下りろ. In the saved results, Qwen’s movement judgment for these two rows differs from the example labels.
2. Switch to the notebook window and open your dh-b-movement.json with 「第2回のワークシートを開く」 (open the class 2 worksheet). If it does not open, the supplied examples become the reference.
3. In the notebook’s 「同じ箇所を分類する」 (classify the same passages), compare each method’s direction, movement judgment, actor, and supporting wording for that passage with your labels, and record the disagreement.
4. In 「段落から探す」 (find in the paragraph), record whether each method found the worksheet expression and what else it extracted.

The notebook calculates agreement between your labels and each method’s output. Agreement shows consistency, not correctness. Seven passages, with rules written specifically for them, cannot establish accuracy on other passages.

For one disagreement, read the paragraph and record which case in the table below applies.

Qwen and Decider received instructions to use the narrator’s perspective, judge whether the movement itself occurs under negation or inability, and treat commands or conditions alone as insufficient evidence that movement happened. The rule-based method classifies by verb ending. 「方法の違い」 (differences between methods) at the end of the notebook summarizes each method.

Keep your worksheet file or notes open beside the notebook: it does not show the criterion you revised in class 2.

uvx --python 3.14 marimo==0.25.0 edit --sandbox movement-extraction.py

Exit note

Record these four items in your notes (.txt or .md) and submit the notes at the submission page.

Homework: decide the question for your class 5 investigation

Allow 20–30 minutes. Submit the result at the submission page before class 5.

In class 5, an agent investigates the same question, and you check its claims with your own commands and the source. Record your own investigation first.

1. Write one question about your author that corpus counts can answer, and your prediction, in the question and prediction fields of the evidence record. The submission page lists the questions and predictions from the submitted records. Example: in this writer’s selected works, does 落ちる combine with 来る more often than with 行く?
2. Run two commands that answer the question, and record the commands, the output counts, and the unit counted (tokens, works, or per 10,000 verbs).
3. Read one passage from the output in its full paragraph, and write one sentence on what the counts show and one on what they do not.
4. Submit the CSV, including rows 11–12, and the evidence record at the submission page before class 5.