NER comparison

Course resources

Compare word-list matching, GiNZA, XLM-RoBERTa, GLiNER2.5, Qwen3.5-4B, and GPT-6.1-Sol Med on the same text. The notebook interface and source text are Japanese.

Open the notebook

Open in your browser (Campus network only)

Run locally

Unpack the notebook bundle (ZIP) and keep all files together, including notebook-theme.css and the hidden .marimo.toml. Run this command from that folder:

uvx --python 3.14 marimo==0.25.0 edit --sandbox ner-comparison.py

If uvx is not recognized, first install uv by following Run a course notebook.

The notebook initially displays saved results. GPT-6.1-Sol Med is a commercial model, GPT-6.1-Sol, run by the instructor with medium reasoning effort; its version is not published. Its results are saved, and this model cannot be run directly in this notebook.

The first extraction downloads several GB of models. The text is processed on the GPU of an Apple silicon Mac and on the CPU elsewhere. No API key is required.

How do I use an NVIDIA or AMD GPU on Linux?

To use an NVIDIA (CUDA) or AMD (ROCm) GPU on Linux, run these commands one line at a time in the same folder instead of the command above:

uv venv --python 3.14
uv export --script ner-comparison.py --no-hashes | grep -v '^torch==' > requirements.txt
uv pip install --torch-backend=auto -r requirements.txt torch==2.14.1
uv run marimo edit --no-sandbox ner-comparison.py

Running the first three lines creates an environment (.venv) in this folder with the package versions in ner-comparison.py.lock and a PyTorch build for your GPU. Afterwards, run only the last line.

Compare results

  1. Inspect each method’s results for part 1 (four paragraphs) of Soranoha’s “The Spider’s Thread”.
  2. Switch to 「蜘蛛の糸：領域別カテゴリ」 (story-specific categories) and compare the extraction results for deities, sinners, and places of punishment. Because word-list matching, GiNZA, and XLM-RoBERTa retain fixed person and location categories, their columns display 「対象外」 (not applicable) and receive no score.
  3. Select 「本文・条件を変えて実行」 (edit text and conditions), change one category name and description, and press 「比較する」 (compare).

You can inspect extraction criteria, provisional reference annotations (the previous GPT-6.1-Sol Med output), and raw outputs inside the notebook. If you edit the text or extraction criteria, the notebook displays extraction results alone, omitting the reference-comparison table.

Word-list matching uses terms supplied for these particular texts. You can edit the terms and categories in 「本文・条件を変えて実行」.

GiNZA and XLM-RoBERTa use NER categories defined during training. The XLM-RoBERTa model was trained on annotated Japanese Wikipedia texts. This comparison uses its person (PER) and location (LOC) categories; other categories remain in the raw output.

GLiNER2.5 is also encoder-based, but accepts category names and descriptions at run time. LLMs accept the same categories and descriptions and generate extraction results. The supplied inputs contain shared category definitions without the names expected in the output.

Check their ability to accept new categories separately from whether their extractions follow your instructions.

The notebook constrains Qwen’s output to JSON with the specified categories. Valid JSON can still omit entities or assign incorrect categories.

Search the course pages