Terms and ideas
Search matches, labeled occurrences, and events
State what you count. The same paragraph can contain several string matches and several descriptions of one event.
…一生懸命にのぼAった甲斐があって…それからあのぼ×んやり光っている…この分でのぼCって行けば…自分ののぼAった後をつけて…よじのぼBって来る…ここへまでのぼAって来た…せっせとのぼBって参ります…
A: Kandata’s climb · B: the sinners’ climb · C: conditional · ×: not movement
-
grep -cF ‘のぼ’: lines - 1 line (5 in the whole file)
-
grep -oF ‘のぼ’: search matches - 7 matches (12 in the whole file)
- Annotation: movement expressions
-
6, excluding
あのぼんやり(×) - Events
-
A judgment: Kandata’s climb (A, described three times) and the sinners’ climb (B, described twice).
のぼって行けば(C) is a condition, not a climb that happened
Text: an excerpt from line 11 of story.txt (one paragraph of part 二), the Soranoha plaintext of 「蜘蛛の糸」.
Before reporting a number, state the unit, included rows or passages, and source version.
How do individual passages change the count?
| Passage | Search match | Labeled occurrence | Event |
|---|---|---|---|
あのぼんやり光っている (part 二) |
Yes: のぼ is part of あのぼんやり |
Not a movement expression | None |
この分でのぼって行けば (part 二) |
Yes | Up; “Not stated whether it happens” (Uu); evidence: conditional ば |
None: Kandata predicts climbing out |
のぼれなくなってしまいました (part 二) |
Yes | Up; “Does not happen / did not happen” (CT-); evidence: のぼれなく |
None: he cannot climb further |
ややしばらくのぼる中に and 一生懸命にのぼった甲斐があって (part 二) |
Yes: each contains のぼ |
Two rows: up, “Happens / happened” (CT+) |
One climb, described twice |
A token is a unit produced by a text analyzer; its boundaries depend on the analyzer and dictionary. Our movement expressions may span several tokens.
When you cannot choose a label
“Not stated whether it happens” (Uu) means the narrator does not state whether the movement happens. Use “Defer judgment” when you cannot choose a label.
Keep the row for that passage. If you cannot choose a label for Direction or “Does the movement happen?”, enter “Defer judgment” in that field and write your reason in Evidence.
| Problem | What to do |
|---|---|
| Not enough context | Read the full paragraph |
| The text does not state who moves | Leave Actor blank; write the wording you checked in Evidence |
| You cannot choose a label using the criteria | Enter “Defer judgment”; write what needs clarification in the criteria |
| You and another reader disagree | Check the evidence for each judgment in the text |
If you exclude rows with deferred judgments from a count, also report how many rows you excluded. If different possible labels change the result, give the count for each case.
Compare independent labels
Compare labels recorded before discussion; keep later revisions separately.
How is inter-annotator agreement measured?
Inter-annotator agreement (IAA) quantifies how consistently annotators apply the same criteria. Raw agreement is matching labels divided by occurrences compared, calculated separately for each field:
| Field | Example |
|---|---|
| Direction | 5 matching labels out of 5: 100% |
| Movement judgment | 3 matching labels out of 5: 60% |
For each field, compare only rows where both readers supplied a label. Report deferrals and unfinished rows separately; two deferrals do not count as matching judgments. Uu is a completed judgment.
Agreement quantifies consistency, not correctness. Five selected examples cannot establish reliability. For larger samples, chance-corrected measures such as Cohen’s κ account for agreement expected from label frequencies; choose a measure suited to the task and annotator count.
Artstein and Poesio (2008), Inter-coder agreement for computational linguistics.