TL;DR — We released ja-doc-classify-v1, a router that sits at the entrance of a Japanese business-document pipeline: it sorts documents into 8 domains, suggests one of 43 document types only when it is confident (and abstains otherwise), and flags documents that may be highly sensitive. On human-written evaluation sets it routes domains at 0.989 accuracy after abstaining on 5–6%, and suggests types at 0.82–0.87 accuracy after abstaining on 26–30%. The publication basis is the human-written evaluation — our synthetic held-out test was deliberately left unopened. Apache 2.0, ONNX included.
- Routing first, labels second. The 8 domains (finance, medical, legal, government, employment, education, e-commerce, telecom/IT) are the routing decision. The 43 document types are candidates, returned only above a confidence threshold; below it, the model returns the domain alone and says the type is unknown. A router that guesses a type for every document poisons everything downstream.
- The numbers are from text the model never trained on, written by people. Two evaluation sets of 200 human-written business documents each, unrelated to the training data: domain accuracy 0.989 on the non-abstained remainder, type accuracy 0.82–0.87 on the non-abstained remainder, and the high-sensitivity flag catches 0.94–0.975 of sensitive documents while flagging 8–12.5% of ordinary ones. CPU: ~141 ms per document (ONNX, Core i7-11700F).
- What we refused to do: the training data is synthetic, and a held-out synthetic test would have produced prettier numbers — synthetic evaluations flatter the model that shares their generator. So the sealed synthetic test stays sealed, and the release stands on the smaller, uglier, human-written measurement instead. The weak types are named in the limitations: contracts, powers of attorney, and identity-verification ledgers scored F1 0 on the full 43-type set.
Where this sits
Document pipelines usually fail at the door. Everything arrives mixed — invoices next to medical certificates next to contracts — and each downstream tool only handles one kind of document well. Our first model, a PII detector, assumes you already know a document deserves scrubbing. This router is the part in front of it: decide what a document is, how sensitive it might be, and only then choose the processing.
Three outputs, three different jobs:
- Domain (8 classes) — the routing decision. High threshold: below 0.95 confidence, the model abstains entirely.
- Document type (43 classes) — a candidate, not a verdict. Below 0.80 confidence, you get the domain only. Expect the type to be checked by a human or a downstream rule wherever it matters.
- High-sensitivity flag — fires when the summed probability of 22 sensitive types reaches 0.12. It is tuned to miss little, at the cost of flagging 8–12.5% of ordinary documents. Treat it as "move this up the human review queue", never as a confidentiality judgment.
Use it
from transformers import AutoModel, AutoTokenizer repo = "japan-opendata/ja-doc-classify-v1" tok = AutoTokenizer.from_pretrained(repo) model = AutoModel.from_pretrained(repo, trust_remote_code=True) model.eval() text = "診断書 在職中、加療を要する状態にあるため、2週間の自宅療養を指示する。" result = model.predict(tok, text) print(result) # {'domain': '医療', 'document_type': '診断書', 'high_risk': True, ...} # When confidence is low, document_type comes back None — domain only. The ONNX build (fp32) returns the three heads' logits; apply the thresholds from the bundled routing_config.json for abstention and the sensitivity flag. PyTorch and ONNX outputs matched on all 600 check items.
Measured results (human-written evaluation)
Two sets of 200 documents each, written independently of the training data (set v1 covers 25 types, v2 all 43), default thresholds applied:
- Domain: abstention 5–6%, accuracy on the rest 0.989 (both sets)
- Document type: abstention 26–30%, accuracy on the rest 0.82–0.87
- High-sensitivity flag: recall 0.94–0.975, false-flag rate on standard documents 8–12.5%
- Speed: ~141 ms per document at the documents' average length (~425 characters), ONNX FP32 on a Core i7-11700F
What didn't work / Limitations
- Some types simply fail. On the full 43-type set, contracts (契約書), powers of attorney (委任状), and identity-verification ledgers (本人確認台帳) scored F1 0 — they get absorbed into similar types. The confusion is mostly between look-alike documents, and the type output must be treated as a candidate for exactly this reason.
- The evaluation is small. 200 documents per set is a measurement, not a census. Real document mixes will shift every number here; the abstention rates will tell you when yours has.
- The synthetic sealed test is unopened, on purpose. Training data is synthetic (built on NVIDIA's Nemotron-PII-Japan with programmatic style expansion into email, short-form, bullet-list, and memo styles). A synthetic holdout would mostly measure how well the model learned the generator. We are not publishing numbers from it, and the release decision rested on the human-written sets instead.
- The sensitivity flag over-fires by design. 8–12.5% of ordinary documents get flagged. That is the price of the 0.94–0.975 recall, and it makes the flag a prioritization signal, not a classification.
- Not a legal instrument. This model must not decide a document's legal nature or whether it may be disclosed. It routes; people decide.
This is the fourth in our line of small, single-purpose Japanese models, and the first designed explicitly as a component for another one: router in front, PII detector behind it. Same rules as the rest: free, ONNX included, measured on data the model didn't train on, failures printed on the card.
Where does document triage happen in your pipeline today — a model, rules on filenames and senders, or a human opening everything?
- Model on Hugging Face: https://huggingface.co/japan-opendata/ja-doc-classify-v1
- Code (the model series): https://github.com/japan-opendata-lab/ja-small-models
- Model page with usage notes: https://jpopendata.jp/models/ja-doc-classify-v1/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).

