TL;DR — We released ja-pii-ner-v1, a 111M-parameter model that detects 18 types of PII in Japanese text and runs fully on-device via ONNX / Transformers.js. On a held-out test of 10,001 documents it scores span F1 0.9770, with zero false positives on 2,068 PII-free documents, at 224 ms per document on a desktop CPU. Trained and evaluated only on synthetic data. Apache 2.0.
- The scrubbing tool should not be another place your raw text leaks to. If you call a cloud API to remove personal data before sending text to an LLM, the unscrubbed text has already left your hands. Running the detector locally — in the browser, or on your own CPU — makes the scrubbing step itself private.
- Measured, not promised: span precision 0.9965 / recall 0.9582 / F1 0.9770 on the full test split (10,001 documents, 43,064 gold spans), and 0 false positives across 2,068 documents that contain no PII at all. The ONNX fp32 export reproduces the PyTorch model exactly on the full test set.
- The honest weak points are in the open: ORGANIZATION recall is 0.879 and SECRET recall is 0.800. Driver's license and passport are not covered by the public test set, so they are unevaluated. Details below, including the INT8 build we measured and decided not to ship.
The problem
Plenty of teams want to scrub Japanese text — support tickets, chat logs, internal documents — before pasting it into an LLM or sharing it outside the team. The common options are a regex pile that misses half the formats, or a hosted redaction API. The API option has a structural flaw: to remove the personal data, you first send all of it, unremoved, to someone else's server.
A small encoder model changes the trade-off. At 111M parameters it runs on a CPU, and with ONNX it runs in the browser through Transformers.js. The text never leaves the device; only the scrubbed output goes anywhere.
The model
- ja-pii-ner-v1 — token classification (NER) for Japanese, based on tohoku-nlp/bert-base-japanese-v3
- 18 entity types: PERSON, ORGANIZATION, ADDRESS, POSTAL_CODE, PHONE, EMAIL, DATE_OF_BIRTH, MY_NUMBER, BANK_ACCOUNT, CREDIT_CARD, DRIVER_LICENSE, PASSPORT, HEALTH_INSURANCE, EMPLOYEE_ID, IP_ADDRESS, URL_PERSONAL, USERNAME, SECRET
- It returns exact character spans. Masking or replacement is up to you; the repository ships a
redact.pyhelper, an ONNX Runtime example, and a Microsoft Presidio recognizer - Apache 2.0, ONNX export included
- No real personal data anywhere in the pipeline. Training data is NVIDIA's synthetic Nemotron-PII-Japan (79,998 documents) plus 25,000 synthetic documents we generated from public registries (postal-code data, the corporate-number registry). Evaluation is on synthetic data too — treat the numbers accordingly
Run it in the browser
import { pipeline } from '@huggingface/transformers'; const ner = await pipeline('token-classification', 'japan-opendata/ja-pii-ner-v1'); const out = await ner('担当は山田太郎(電話 090-1234-5678、メール taro.yamada@example.com)です。'); console.log(out); That is the whole integration for a first pass: the model loads from the Hugging Face repo and inference happens on the user's machine. For production span handling — merging subword pieces back into character spans, chunking documents longer than 512 tokens — the repository's helpers are the reference implementation. A hosted browser demo is on the way.
Measured results
Test set: the full test split of Nemotron-PII-Japan converted to character spans (10,001 documents, 43,064 gold spans), plus a separately generated no-PII set of 2,068 documents. Threshold-free BIO argmax decoding; no tuning was done against the test set.
- Precision (micro): 0.9965
- Recall (micro): 0.9582
- F1 (micro): 0.9770
- False positives on PII-free documents: 0 / 2,068
- CPU speed (Intel i7-11700F, ONNX fp32): 224 ms per document, vs 814 ms for PyTorch fp32
Pattern-based labels (POSTAL_CODE, EMAIL, MY_NUMBER, BANK_ACCOUNT, CREDIT_CARD, and several others) sit at or near F1 1.0000. The drag is where you would expect it: names of organizations that look like person names, and secrets in unusual formats.
For context, we also ran the closest comparable public model (az225/privacy-masker-ja-pii-ner-30m) on the identical test set. On the three labels that map 1-to-1 (PERSON, ORGANIZATION, ADDRESS), ours scores F1 0.9626 against 0.6568 — with a caveat that matters: the label definitions differ (az225 splits place names into a separate LOCATION label, and its PERSON recall of 0.9961 is higher than ours at 0.9577). Part of that gap is a definitions gap, not only a quality gap. The full per-label table is on the model card.
What didn't work / Limitations
- INT8 quantization is not shipped. We measured it: dynamic INT8 cut the model to 188 MB and 122 ms per document, but micro F1 fell to 0.8988, with large recall losses exactly on the numeric and symbol-pattern labels (IP_ADDRESS, URL_PERSONAL, SECRET, BANK_ACCOUNT). Static INT8 degraded the model entirely. fp32 is the supported configuration.
- ORGANIZATION recall (0.879) and SECRET recall (0.800) are the weakest points. Organization names that read as person names, and secrets in unusual formats, are the typical misses.
- DRIVER_LICENSE and PASSPORT are unevaluated. The public test set contains no gold spans for them; they are trained from our synthetic data only. Do not read the headline F1 as covering them.
- Synthetic in, synthetic out. All training and evaluation data is synthetic. Performance on real documents will differ, and these numbers are not a guarantee of complete detection. The source dataset is also known to contain some missing gold labels, so a small part of the measured false positives may be unlabeled true PII.
- 512-token truncation. Inputs longer than 512 tokens are truncated by the model itself; use the chunking in
redact.pyfor long documents.
Where this goes
This is the first in a line of small, single-purpose Japanese models we plan to publish the same way: free, ONNX included, measured on a public test set with the failures printed on the card. Parts, not products — components you can drop into your own pipeline.
If you scrub Japanese text before it reaches an LLM today, what do you use — regex, a hosted API, or nothing yet?
- Model: https://huggingface.co/japan-opendata/ja-pii-ner-v1
- Model page with usage notes: https://jpopendata.jp/models/ja-pii-ner-v1/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).
