Class 3: Searching by lemma and pattern

1. Set up the prepared corpora

On the computer you use, install the search tool ling (lingrep 0.5.0) once by following Set up ling.

1. Save the prepared corpora (ZIP, about 18 MB) into your class 2 working folder dh-b-2026 and extract it.
2. Open a terminal and run these commands one line at a time.

cd dh-b-2026/class3-prepared
ling --version
ling corpus inspect story-unidic-novel.lingcorpus

A prepared corpus (.lingcorpus) is a text whose analysis by an analyzer and a dictionary has been saved. ling reads it with --corpus and does not require a dictionary.

If inspect reports one document and finishes with complete:, continue to the next section. If the command is unavailable, follow Set up ling and run ling --version again.

ling doctor
ling matches --pattern '[lemma=上る].' --context 2 --color never story.txt

2. Widen a string search into a pattern

Class 2 used -F, which matches the exact string. A regular expression is a search pattern in which symbols stand for any one character or for one of several characters. With -E, grep reads its argument as a pattern.

grep -oE 'のぼ.' story.txt | sort | uniq -c
grep -oE 'のぼ[っりるれ]' story.txt | wc -l

A pipe (|) makes the output of the command on its left the input of the command on its right. The diagram below extends the first command with sort -rn | head from section 8 and shows the output after each stage.

のぼん is part of あのぼんやり, which is not a movement expression. The second command yields 11 matches, excluding のぼん. よじのぼっ is counted among the 11 as のぼっ. To separate and count compound verbs, use the analysis in the next section.

3. Read the UniDic analysis and search by lemma

UniDic is a dictionary for morphological analysis published by the National Institute for Japanese Language and Linguistics. The analyzer splits the text into short-unit tokens and gives each token a lemma, a part of speech, and an inflected form. Today’s main dictionary is 近現代口語小説UniDic, the variant prepared for Meiji and Taishō fiction.

A lemma (UniDic 語彙素) is the dictionary headword that groups inflected forms and spelling variants; it is the lemma column in the output. のぼっ, のぼり, のぼる, and のぼれ share the lemma 上る, while よじのぼっ belongs to the compound verb 攀じ登る.

ling tokenize --corpus story-unidic-novel.lingcorpus --tsv | less

TSV means tab-separated values. The same table is in tokens-unidic-novel.tsv.

In less, type / followed by a string to search, and q to quit. Find のぼっ and よじのぼっ and check their lemma and part of speech.

ling check --pattern '[lemma=上る].'
ling matches --corpus story-unidic-novel.lingcorpus --pattern '[lemma=上る].' --context 2 --color never

check verifies the syntax, and matches shows each match with two tokens of context on each side. In the distributed story.txt with today’s dictionary, expect 10 token matches.

Class 2’s grep -oF 'のぼ' returned 12 matches, the pattern in section 2 returned 11, and the lemma search returned 10. Explain in your notes how あのぼんやり and よじのぼっ account for the differences among these counts.

4. Read the same text under other analyzers and dictionaries

The dictionary defines the lemma. The ZIP archive contains the same text analyzed by these six combinations.

Four surface forms from the text are analyzed across the corpora as follows:

A lemma value depends on the dictionary’s definitions, so a pattern written for one dictionary does not carry over to another. In this text the three UniDics return the same lemmas, but in another text the dictionary’s period and register change the results.

1. Run the [lemma=上る] pattern from section 3 on story-ginza and story-spacy as well and compare the counts. In the distributed story.txt they are 10, 0, and 2.
2. Which analyzer and dictionary would you choose to count 攀じ登る and 這い上る each as one verb? Write your reason in your notes.

ling summarize --corpus story-unidic-novel.lingcorpus --patterns compare.lx --top all
ling summarize --corpus story-ginza.lingcorpus --patterns compare.lx --top all
ling summarize --corpus story-spacy.lingcorpus --patterns compare.lx --top all

5. Find the movement that string search missed

The class 2 and homework searches found only the places containing のぼ and 落ち. With lemmas and parts of speech you can count every verb in the text. In class3.lx, each name := pattern. is one definition, key: [lemma] makes the lemma the unit of counting, and lines starting with # are notes.

less class3.lx
ling check --patterns class3.lx
ling summarize --corpus story-unidic-novel.lingcorpus --patterns class3.lx --top all > summary.txt
less summary.txt

The verbs table displays verb lemmas sorted by frequency. In the distributed story.txt, 上る and 言う each occur 10 times, and 落ちる and 光る each occur 5 times. vertical is a set of 12 lemmas chosen by the instructor. 垂れる and ぶら下がる denote the physical state of the thread and of Kandata.

1. From verbs, write down every lemma that expresses vertical movement, and mark those that the のぼ and 落ち searches could not find.
2. Compare your list with vertical and choose one lemma to add or one to remove. Decide with the rule you revised in class 2 whether 垂れる and ぶら下がる count as movement expressions.
3. Change the set in class3.lx, rerun check and summarize, and record why you changed it.

6. Find the word forms that cue the judgments

In class 2 you judged “Does the movement happen?” by reading the paragraph. Imperative, conditional, and negated forms are cues for that judgment.

ling matches --corpus story-unidic-novel.lingcorpus --pattern '[unidic.cform ~ 命令].' --context 3 --color never
ling matches --corpus story-unidic-novel.lingcorpus --pattern '[unidic.cform ~ 仮定].' --context 3 --color never
ling matches --corpus story-unidic-novel.lingcorpus --pattern '[upos=VERB] [lemma in {ない, ぬ}].' --context 3 --color never

A word form is a cue, not the judgment itself. 落ちてしまわなければなりません matches the conditional, yet the narrator presents the fall as unavoidable. unidic.cform exists only in UniDic output and is unavailable in the other corpora of section 4.

1. Keep only the matches that are movement expressions and compare them with your worksheet judgments.
2. Choose one passage with no imperative, conditional, or negated form (most of “Happens / happened” (CT+)) and write down what evidence its judgment rested on.

7. Read the narrator’s viewpoint from 来る and 行く

After a movement verb, 来る and 行く indicate the narrator’s viewpoint on the movement.

ling matches --corpus story-unidic-novel.lingcorpus --pattern '[upos=VERB] [lemma=て] [lemma in {来る, 行く}].' --context 3 --color never

In the distributed story.txt there are 7 matches. The deixis table in summary.txt records counts for each combination of verb and 来る or 行く.

1. For each match, record who moves and whether it is told with 来る or 行く.
2. Which word is used for Kandata’s climb, and which for the sinners’ climb? Where is the viewpoint in part 二? Compare this with the actor column of your worksheet and the narrator’s-perspective judgment.

8. Widen to 103 works by Akutagawa

akutagawa-unidic-novel.lingcorpus is a prepared corpus of 103 works by Akutagawa Ryūnosuke from Soranoha, analyzed with the same dictionary. akutagawa-works.csv records Soranoha’s titles and first-publication information, with source links to the work information and reader links to the text. Run the same pattern file and check whether the observations from one story hold in the other works.

ling summarize --corpus akutagawa-unidic-novel.lingcorpus --patterns class3.lx --top all > akutagawa-summary.txt
grep vertical akutagawa-summary.txt
grep deixis akutagawa-summary.txt | grep -E '上る|落ちる|下りる'

grep pulls from the saved summary only the rows that contain a table name or a lemma.

Counts by work come from the tabular output of matches.

ling matches --corpus akutagawa-unidic-novel.lingcorpus --tsv --patterns class3.lx | grep -w vertical | cut -f3 | sort | uniq -c | sort -rn | head
ling matches --corpus akutagawa-unidic-novel.lingcorpus --where '@document.title = 杜子春' --pattern '[lemma in {上る, 落ちる, 下りる}].' --context 4 --color never

The diagram below traces the output after each stage of the first command.

In 「杜子春」, 登って来た matches the lemma 上る. The lemma also groups kanji spellings, so a string search for のぼ cannot find this passage. The counts over 103 works depend on the length of each work, its year, and the dictionary’s decisions.

1. Choose one work with many matches, show its matches with --where, and read two passages in full paragraphs.
2. Classify them with the same three judgments as for 「蜘蛛の糸」 and record whose viewpoint 来る and 行く mark.
3. Write one claim the counts support and one they cannot establish.

Exit note

Record these five items. You will also use class3.lx in class 4 on prepared corpora of other authors, so keep the class3-prepared folder and summary.txt in your working folder.

Homework: record three passages you found as CSV rows 8–10

Allow 20–30 minutes. The class 2 worksheet is limited to seven rows, so add the three passages to the CSV. You will extend this CSV in class 4 (rows 11–12) and use it for the class 5 evidence assignment.

1. Save the seven worksheet rows with example labels (CSV) into your working folder and open it in a plain-text editor.
2. Choose one movement lemma from section 5, one match from section 6, and one match from section 7 or 8, and read the full paragraph of each passage. For a section 8 passage, write the work’s title in part.
3. Add the three passages as rows 8–10 with the same columns and the values in the table below.
4. Note the corpus, pattern, and count used for each row. Submit the CSV, your saved worksheet file dh-b-movement.json (if you attended class 2), and the notes before class 4 at the location in the submissions table. In class 4, you will compare the labels for rows 2–7 on the submission page.

Classification reference: column meanings and judgment criteria.