Agenda

Aozora Bunko

Aozora Bunko abridged history

• volunteer-driven digital library (1997–) [@AozoraBunko]
• currently >14,000 out-of-copyright and copyright-free works written by over 1000 different authors
• long/short works of fiction and non-fiction, various genres
• large ecosystem of surrounding tools; widely read with differeny ereader products; name of Aozora Bunko transcends project to mean out-of-copyright works in general [@大久保ゆう2017en]

• traditionally employs a split between transcriber and proofreader roles

Significance

GitHub and infrastructure

• site and (not all) data mirrored in a version-controlled repository on GitHub, updated daily
• development happens on different infrastructure (includes automated proofreading checkers, etc.)

Issues

• text files are bundled with image files in zip files
  • binary files not ideal for revision control
• HTML conversions are kept as plaintext
  • (X)HTML contains structural tags and limited metadata: DC.Title, DC.Creator, DC.Publisher.
• (however,) the official format is the text file in so-called “Aozora Bunko format”
• repository contains a lot of binary files and tends to contain changes that are hard to track effectively with git
• 14GB in size (!) for ~100m token text corpus

Research aims

1. The present work aims to develop the infrastructure to be able to provide continuously updated plaintext and TEI versions of works in the Aozora Bunko.
2. Automatic generation of Linked Open Data from Aozora Bunko metadata to enable rich faceted search of this large dataset with applications to dataset construction facilitating diachronic/genre/authorship attribution/stylistics, and similar studies.
3. Expose the resource to the wider non-Japanese speaking research community.

Text Processing

Japanese language text processing

• complex writing system: romaji (roman characters), hiragana, katakana, Chinese characters (kanji)
  • multiple standards of word units sizes (word segmentation is an issue); requires preprocessing in the form of morphological analysis
• text encoding issues
  • SJIS (JIS X 0208) as default encoding of Aozora Bunko
  • characters from the 3rd and 4th planes of the JIS X 0213 are mapped as textual descriptions and image files
    • ※(「韋＋備のつくり」、第3水準1-93-84)
    • “combination of two characters 韋 and 備, corresponds to 1-93-84 on the 3rd plane of JIS X”
  • conversion to Unicode using JIS X mapping table necessary

Annotations included and missing in Aozora Bunko

• section/chapter/…, paragraph boundaries provided for using ［＃ ... ］ blocks
  • sentence boundaries need to be detected automatically
• direct speech, quotations, kanji katakana majiri bun (use of Chinese characters with katakana instead of hiragana)
• furigana (ruby) is tagged next to Chinese characters between 《...》 blocks
• clear separation between main prose and afterword, foreword, etc.
  • sometimes untagged even in official format–problem for authorship attribution
• morphological analysis for word segmentation and part-of-speech tagging

Natural Language Toolset

• Morphological preprocessing: MeCab [@kudo2004applying]
• National Institute of Japanese Linguistics’ UniDic dictionary chosen based on the age and writing style of each work [@OkaUnidicCWJ220en]
  • Contemporary Written Japanese (CWJ) UniDic 2.3.0
  • Modern Japanese (Kindai) UniDic 1603

Text processing with the Aozora Corpus Generator (1)

• uses XPath and regular expressions to filter out non-main body text from the HTML version of source files
• converts JIS X 0213 characters into Unicode

Text processing with the Aozora Corpus Generator (2)

• kanji katakana majiri bun (use of Chinese characters with katakana instead of hiragana) detected: conduct morphological processing by temporarily converting all katakana to hiragana
• direct speech removal
• filtering by token count

• predominantly used for corpus lingustics type analyses, not focused on metadata

LOD

Linked Open Data: Previous efforts

• @AozoraBunkoLOD’s “Aozora Bunko Linked Open Data” from 2012
• uses Aozora Bunko’s bibliography CSV file to create LOD

<http://www.aozora.gr.jp/cards/001257/card46658.html> a
    aozora:BibResource .
<http://www.aozora.gr.jp/index_pages/person1257.html> a
    aozora:Person ,
    foaf:Person ;
    aozora:authorID "1257"^^xsd:int ;
    foaf:familyName "アーヴィング" ;
    foaf:givenName "ワシントン" ;
    aozora:familyNameTranscription "アーヴィング" ;
    aozora:givenNameTranscription "ワシントン" ;
    aozora:familyNameForSort "ああういんく" ;
    aozora:givenNameForSort "わしんとん" ;
    aozora:familyNameInRomaji "Irving" ;
    aozora:givenNameInRomaji "Washington" ;
    rdag2:dateOfBirth "1783-04-03"^^xsd:date ;
    rdag2:dateOfDeath "1859-11-28"^^xsd:date ;
    foaf:name "アーヴィング ワシントン" ;
    rdfs:seeAlso <http://id.ndl.go.jp/auth/ndlna/00444295> , ...

LOD: Problems

• unprocessed dates as strings

  “1957（昭和32）年5月20日” ↓

  “1957(32nd year of Shōwa period)Y 5M 20D”

  should be "1957-05-20"^^xsd:date

_:node17fc51317x3 dcterms:title "スケッチ・ブック" ;
    aozora:firstIssued "1957（昭和32）年5月20日" ;
    aozora:inputVersion "2000（平成12）年2月20日33刷改版" ;
    aozora:modifiedVersion "2000（平成12）年 2月20日33刷改版" ;
    dcterms:publisher "新潮文庫、新潮社" .

• also includes textual descriptions such as 紀元前6世紀初 (beginning of the 6th century BC), 「太陽　創刊号」 (first publishing of the magazine The Sun = July 1963)

LOD: Current solutions (1)

• can use git to access the history of the Aozora Bunko repository and pull updates
• currently only generates LOD, with TEI corpus export due later (will discuss in TEI section)

LOD: Current solutions (2)

<http://abc.com/0.1/w050367>
    abc:aozoraLastModifiedDate
        "2014-09-21"^^xsd:date ;
    abc:aozoraPublishingDate
        "2010-06-21"^^xsd:date ;
    abc:author    "p001263" ;
    abc:bibResource
        <https://www.aozora.gr.jp/cards/001263/card50367.html> ;
    abc:copyrightExpired
        false ;
    abc:firstPublished
        "1933-09-01"^^xsd:date ;
    abc:ndc       [ abc:ndc/category "中国" ] ;
    abc:ndc       [ abc:ndc/category "朝鮮" ] ;

*abc.com is a placeholder

LOD: Current solutions (3)

    abc:orthographicStyle
        "新字新仮名" ;
    abc:references
        <http://abc.com/0.1/黒船前後・志士と経済他十六篇> ;
    abc:revisionSource
        "1981（昭和56）年7月16日第1刷" ;
    abc:revisor   "小林繁雄" ;
    abc:sources   <https://www.aozora.gr.jp/cards/001263/files/50367_39396.html> , <https://www.aozora.gr.jp/cards/001263/files/50367_ruby_38317.zip> ;
    abc:subtitle  "はつりょうえんせいたい" ;
    abc:title     "撥陵遠征隊" ;
    abc:transcriber
        "ゆうき" ;
    abc:transcription
        "はつりょうえんせいたい" ;
    abc:transcriptionSource
        "1981（昭和56）年7月16日第1刷" .

TEI Encoding

TEI Header Mapping (1)

fileDesc

• titleStmt: title, author, editor
• extent: character, morpheme, sentence, paragraph, section counts (planned)
• publicationStmt: publisher, distributor, availability, date, idno
• sourceDesc, encodingDesc, projectDesc
• respStmt: resp, name

TEI Header Mapping (2)

profileDesc

• langDesc
• textClass: classCode (mapped to Nippon Decimal Classification)
• textDesc: (n: novel, letter, …), channel, constitution, derivation (original vs. translation), factuality (fiction vs. non-fiction) , purpose

revisionDesc

• who/when: two actors—Aozora Bunko and current system

TEI Body Mapping

• PEGJS grammar (~300 lines) provided by AozoraHack community converted to EBNF
• initial focus will be on providing structural tags and quotation support (direct speech detection ported over from Aozora Corpus Generator)
• morphological processing: morpheme (Short Unit Word) boundaries, and translated part-of-speech tagset

Discussion

Discussion

• Aozora Bunko text encoding format and distribution tailored to transcribers/proofreaders and ebook readers, respectively
  • more work needed to make machine readable metadata and corpus files useful for lingustic analyses
• reconciling TEI and Open Linked Data approaches to certain aspects of metadata encoding
  • current vocabulary not aligned (TEI pointers?)

Future Work

Future Work

• upload git repository containing LOD and TEI
• Universal Dependencies [@UniversalDependencies] part-of-speech tag mapping from UniDic
• filename convention for easier import in tools such as Stylo [@eder2016stylometry]
• stand-off annotation
  • “patches” to fix issues uncovered in certain texts before they are upstreamed in Aozora Bunko itself
• git “archaeology” to uncover Aozora Bunko trends