Introduction

Situating Aozora Bunko in relation to other corpora

References: Meiroku Zasshi: @近藤明日子2012明六雑誌コーパス; Kokumin no Tomo: @近藤明日子2016明六雑誌コーパス; The Sun: @田中牧郎2005言語資料としての雑誌; Kindai Josei Zasshi: @田中牧郎2006近代女性雑誌コーパス; Balanced Corpus of Contemporary Written Japanese: @Maekawa2013

Aozora Bunko metadata schema

Official database schema for Aozora Bunko’s author-work CSV file copied from https://www.slideshare.net/takahashim/osc201703-aozorahack.

Interim solution: Aozora Bunko Corpus Generator

• Aozora Bunko Corpus Generator is a Python preprocessing script that generates plain and tokenized text files from the Aozora Bunko for use in corpus-based studies.
• Works on the official GitHub repository of Aozora Bunko, and specifically on the HTML versions of works (*).
• Can specify specific authors and works to extract, or can extract all data.
• Basic punctuation/symbol removal, morphological feature extraction (surface form or lemma, or a combination of any MeCab dictionary field), filtering of works smaller than specified token threshold.
• Outputs a CSV of metadata.

(*) This is a slight issue.

Morphological processing issues

• At this point, morphological processing issues come down to choice of dictionary:
  • IPAdic 2.7.0
  • UniDic 2.1.0
  • UniDic CWJ 2.2.0
  • UniDic CSJ 2.2.0
  • Kindaigo (Modern Japanese) UniDic 1603
• At first glance, the Kindaigo Modern Japanese dictionary might seem like the right choice…

References: @UniDic; @OkaUnidicCWJ220en

Japanese novels corpus

• All novels are extracted from the HTML files of Aozora Bunko using aozora-corpus-generator.
• All metadata is included in the “groups.csv” file. Parts (brow and genre) are sourced from the manually specified ‘author-title.csv’ in the aozora-corpus-generator repo, while the rest are extracted from the master list in the Aozora Bunko repository (list_person_all_extended_utf8.zip).
• An additional non-free part (5 works of Yasunari Kawabata) is then added to supplement the high-brow part.

Japanese novels corpus: Issues

• “Gaiji” characters with provided JIS X 0213 codepoints are converted to their equivalent Unicode codepoint. Aozora Bunko is conservative in encoding rare Kanji, and, therefore, uses images (html version) or textual descriptions (plaintext version).
• Words are sometimes emphasized in Japanese text with dots above characters, while Aozora Bunko uses bold text in their place. Emphasis tags are currently stripped.
• Not all footer information is marked with metadata. Currently several heuristics are in place to detect these occurrences (Example regular expression: '^[　【]?(底本：|訳者あとがき|この翻訳は|この作品.*翻訳|この翻訳.*全訳)').

Japanese novels corpus overview

High group (6 authors, 25 works)

• Natsume Soseki (4)
• Dazai Osamu (2)
• Mori Ogai (3)
• Kobayashi Takiji (4)
• Tanizaki Jun’ichiro (6)
• Yasunari Kawabata (6) (not Aozora Bunko)

Low group (6 authors, 22 works) (cf. Proletariat literature)

• Yumeno Kyusaku (3)
• Edogawa Ranpo (4)
• Unno Juza (4)
• Okamoto Kido (5)
• Oguri Mushitaro (3)
• Sakaguchi Ango (3)

Japanese novels corpus: 夏目漱石 Natsume Soseki (1867–1916) HIGH

NDC: 小説、物語 Fiction

• 草枕 Kusamakura
• こころ Kokoro
• 三四郎 Sanshirou
  • 「ハイドリオタフヒア」
• 坊っちゃん Bocchan

Japanese novels corpus: 太宰治 Dazai Osamu (1909–1948) HIGH

NDC: 小説、物語 Fiction

• 人間失格 Ningenshikkaku
• 斜陽 Shayou

Comments:

• Sakaguchi Ango: possibly friends; dark subject matter

Japanese novels corpus: 坂口安吾 Sakaguchi Ango (1906–1955) LOW

NDC: 小説、物語 Fiction

• ジロリの女 Jirorinoonna
• 復員殺人事件 Fukuinsatsujinjiken
• 不連続殺人事件 Furenzokusatsujinjiken

Japanese novels corpus: 夢野久作 Yumeno Kyusaku (1889–1936) LOW

NDC: 小説、物語 Detective

• ドグラ・マグラ Dogura Magura
  • Kyushu dialect: 「……イヤラッサナア……マアホンニ……タマガッタガ……トッケムナカア……ゾウタンノゴト……イヒヒヒヒヒ……」
  • 「ウーム。ナルホド。ウーム」
• 少女地獄 Shoujojigoku
• 押絵の奇蹟 Oshienokiseki

Japanese novels corpus: 江戸川乱歩 Edogawa Ranpo (1894–1965) LOW

NDC: 小説、物語 (児童書) Detective

• 魔法博士 Mahouhakase
• 少年探偵団 Shounentanteidan
• 宇宙怪人 Uchuukaijin
  • 『キミ、ニゲル、ハイニナル。』
• 灰色の巨人 Haiironokyojin
  • 「……汽車アオーラアーイ」

Japanese novels corpus: 海野十三 Unno Juza (1897–1949) LOW

NDC: 小説、物語 (児童書あり) SF/Young SF

• 浮かぶ飛行島 Ukabuhikoutou
  • アレモ人ノ子。生キテイル。
• 怪塔王 Kaitouou
• 火星兵団 Kaseiheidan
• 地球要塞 Chikyuuyousai

Japanese novels corpus: 岡本綺堂 Okamoto Kido (1872–1939) LOW

NDC: 小説、物語 Horror (怪奇)/Detective

• 半七捕物帳: 69 白蝶怪 Hanshichitorimonochou: 69 Hakuchoukai
• 青蛙堂鬼談 Seiadoukidan
• 飛騨の怪談 Hidanokaidan
• 両国の秋 Ryougokunoaki
• 玉藻の前 Tamamonomae

Japanese novels corpus: 森鴎外 Mori Ogai (1862–1922) HIGH

NDC: 小説、物語 Fiction/Non-Fiction

• ヰタ・セクスアリス Uita Sekusuarisu
• 青年 Seinen
• 渋江抽斎 Shibuechuusai (Non-Fiction)

Japanese novels corpus: 小栗虫太郎 Oguri Mushitaro (1901–1946) LOW

NDC: 小説、物語 Horror (怪奇)

• 黒死館殺人事件 Kokushikansatsujinjiken
• 白蟻 Shiroari
• 潜航艇「鷹の城」 Senkoutei “Habihitsuburugu”

Comments:

• Oguri and Unno were friends.

Japanese novels corpus: 小林多喜二 Kobayashi Takiji (1903–1933) HIGH

NDC: 小説、物語 Fiction

• 蟹工船 Kanikousen
  • 漢字カタカナ交じり文：“ヿ” koto → ………僕ハ今年カラ、今日マデ日記ニ記スコトヲ躊躇シテイタヨウナ事柄ヲモアエテ書キ留メルヿニシタ。
• 工場細胞 Koujousainou
  • ――ダイジョウブカ。
  • ――レッコ、レッコ！
• 党生活者 Touseikatsusha
• 不在地主 Fuzaijinushi

Japanese novels corpus: 谷崎潤一郎 Tanizaki Jun’ichiro (1886–1965) HIGH

NDC: 小説、物語 Fiction

DBpedia entry

• 鍵 Kagi
  • 分ケテモスカートノ下カラ踝ノ辺ノ歪曲美ニ見惚レナガラ、今夜ノヿヲ考エテイタ。……
• 春琴抄 Shunkinshou
• 少将滋幹の母 Shoushoushigemotonohaha
• 痴人の愛 Chijinnoai
• 卍 Manji
• 盲目物語 Moumokumonogatari

Japanese novels corpus: 川端 康成 Yasunari Kawabata (1899–1972) HIGH

NDC: 小説、物語 Fiction

• 伊豆の踊子 Izu no Odoriko
• 古都 Koto
• 千羽鶴 Senbazuru
• 山の音 Yama no Oto
  • 「千二ヨ、タズネルモノハ、コノサキニアル。ワレワレハ、ナカデマツ」
  • 「バカヤロー。キチガイ！」
  • 「ナムアムダブツ。」
  • 三、姉ト弟トハ徳光光子ノ愛情ガ第三者ニ移ルコトナ…
• 眠れる美女 Nemureru Bijo
• 雪国 Yukiguni

Comments:

• Influenced by Tanizaki Jun’ichiro (DBpedia)

Lexical complexity measures

• l is text length (tokens)
• v is vocabulary size (types)

ttr        = v/l
guiraud_r  = v/sqrt(l)
herdan_c   = log(v)/log(l)
dugast_k   = log(v)/log(log(l))
dugast_u   = log(l)^2/(log(l) - log(v))
maas_a2    = (log(l) - log(v))/log(l)^2
tuldava_ln = (1 - v^2)/(v^2 * log(l))
brunet_w   = l^(v^-0.172)
cttr       = v/sqrt(2 * l)
summer_s   = log(log(v))/log(log(l))

Frequency spectrum and vocabulary size-based measures

• fs is frequency spectrum (histogram of frequencies)
• hapaxlegomena is number of tokens with frequency 1
• dislegomena is number of tokens with frequency 2

sichel_s = dislegomena/v
michea_m = v/dislegomena
honore_h = 100.0 * log(l)/(1 - hapaxlegomena/v)
ent = sum(freq_size * -log(freq / l) * (freq / l) for (freq, freq_size) in fs)
yule_k = 10000 * (sum(freq_size * (freq / l)^2 for (freq, freq_size) in fs) - (1 / l))
simpson_d = sum((freq_size * (freq / l) * ((freq - 1) / (l - 1)) for (freq, freq_size) in fs))
herdan_vm =  sqrt(sum(freq_size * (freq / l)^2 for (freq, freq_size) in fs) - (1 / v))
hdd(l, fs, sample_size=42) = sum((1 - dhyper(0, freq, l - freq, sample_size)) / sample_size for (word, freq) in fs)

Authorship attribution

• Extracted a new dataset from the Japanese novels corpus. Used the features flag to append the part-of-speech to each token. Example:

東京＜名詞＞
の＜助詞＞
まん中＜名詞＞
に＜助詞＞
ある＜動詞＞
有名＜形状詞＞
[...]

• Used three distance measures (euclidean, cosine, KL) to cluster the works according to their length-normalized use of particles.
• All work here by Uesaka Ayaka.

Authorship attribution (Euclidean distance)

Authorship attribution (Cosine distance)

Authorship attribution (KL distance)

Topic model on Dec. 2017 version of Aozora Bunko

• Gensim topic model (50 topics, α=auto, passes=15): [Interactive visualization link]

Topic model on Jan. 2018 version of Aozora Bunko

• Uses newer UniDic CWJ 2.2.0; newer version of Aozora Bunko; includes better handling of kanji-katakana-majiri writing style; other small bugfixes: [Interactive visualization link]

Discussion points

• Balance issues:
  • high and low
  • genre
  • length
  • number of works per author
  • variation within works by author
• Dictionary choice:
  • kanji-katakana-majiri
  • spoken (dialogue) vs written
  • leaning toward the new UniDic CWJ 2.2.0 (and also perhaps a custom dictionary for issues uncovered by tokenization comparisons)