How SpotRecall picks the right translation direction for kanji-only queries on-device — using two character tables generated from national standards, and a simple rule for the truly ambiguous case.

SpotRecall, my private on-device browser recall extension, searches across languages by translating the query. Type air conditioner installation and it also searches the Japanese and Chinese versions of that phrase, so a Japanese product page you read last week can come back from an English query.

A translation needs a source language, and the quality of the result depends on getting it right. For most text that is easy. For one important case — queries written only in Han characters — it is a genuinely interesting problem, and SpotRecall 1.3.1 solves it in a way that runs entirely on-device, needs no extra model, and fits in two generated strings.

The interesting case: Han-only text

Chinese and Japanese share a large set of characters. A query like 会社概要 ("company overview") or 名片管理 ("business card management") is written entirely in characters both languages use. Read character by character, it could belong to either.

Statistical language detectors find this hard. Short, kanji-only Japanese tends to be labeled Chinese, because Han characters are far more frequent in Chinese text. And in Chrome specifically, LanguageDetector is an optional on-device download that is not always present, even on machines where Translator works.

So 1.3.1 changes the order of operations: read the script first, and use a detector only when the script cannot decide.

Step one: most scripts decide immediately

Before any statistics, a few rules settle most queries:

any hiragana or katakana → Japanese 表札の通販 any Hangul → Korean 문패 otherwise Latin letters → English exercise bike only Han characters → needs a closer look 

A single kana character is enough: Chinese never uses it. That covers almost all natural Japanese queries, because particles like の and が are written in kana. What remains are Han-only queries: short noun phrases, place names, product categories, and Chinese text.

Step two: characters only one language uses

Even when two languages share most of their characters, each has characters the other does not use. Simplified Chinese writes 们, 语, 译, 门, and 车. Japanese writes 帯, 図, 飯, 訳, and 駅 — forms that simplified Chinese replaced.

Rather than curate such a list by hand, SpotRecall generates it from the two national character-set standards. GB2312 defines the common characters of Simplified Chinese; JIS X 0208 does the same for Japanese. Each has a "level 1" of everyday characters:

zh-only = GB2312 level 1 minus all of JIS X 0208 → 1,385 characters (们 语 译 门 车 …) ja-only = JIS X 0208 level 1 minus all of GB2312 → 1,085 characters (帯 図 飯 訳 駅 …) 

Characters present in both standards — 日, 本, 中, 国 — appear in neither table, so they never sway the decision.

The generator needs no data files. Node's built-in TextDecoder already knows both encodings, so the script simply decodes every two-byte code point in each standard's level-1 range and compares the sets:

function decodeRows(encoding, hiFrom, hiTo) { const dec = new TextDecoder(encoding); const out = new Set(); for (let hi = hiFrom; hi <= hiTo; hi++) { for (let lo = 0xa1; lo <= 0xfe; lo++) { const ch = dec.decode(new Uint8Array([hi, lo])); if (ch.length === 1 && /\p{Script=Han}/u.test(ch)) out.add(ch); } } return out; } const gbLevel1 = decodeRows('gb2312', 0xb0, 0xd7); const gbAll = decodeRows('gb2312', 0xb0, 0xf7); const jisLevel1 = decodeRows('euc-jp', 0xb0, 0xcf); const jisAll = decodeRows('euc-jp', 0xb0, 0xf4); 

The output is a generated TypeScript file with two string constants — about 2,500 characters in total, which is tiny next to a language model.

At query time, SpotRecall counts how many characters of each kind appear:

表札 二世帯 → 帯 is Japanese-only → Japanese 飯能市 地図 → 飯 and 図 are Japanese-only → Japanese 门牌 网购 → 门 and 购 are Chinese-only → Chinese Gardening的日语翻译 → 语 and 译 are Chinese-only → Chinese 

The last example shows the rule at work on mixed input: a Latin word, a Chinese particle, and Chinese characters. The two distinguishing characters are enough to identify the query as Chinese.

Step three: when it is genuinely both, search both

Some queries contain no distinguishing character at all. 会社概要 and 名片管理 are written identically in Chinese and Japanese.

If a detector is available and confident, SpotRecall uses its answer. If not, it does not guess. It translates the query from Chinese and from Japanese, and searches all of the results.

That costs one extra on-device translation. In return, whichever reading the user meant is among the variants. The fusion stage (Post 11) then sorts it out: translations already count at 0.7× the user's own words, and the variant that actually matches pages in the index produces the strong first-place hits that a steep fusion curve rewards.

The full precedence is short:

1. kana → ja, Hangul → ko 2. Han-only: count zh-only vs ja-only characters; the larger count decides 3. no difference: a confident detector result, if available 4. otherwise: translate from both zh and ja 5. Latin with no detector result → en 

Every step is pinned by a test, including the order itself: when the character tables and the detector disagree on Han text, the characters win. 表札 二世帯 stays Japanese even if a detector reports Chinese, because 帯 settles the question.

What it measured

On the evaluation set — 54 mixed Chinese, Japanese, and English queries against a 1,088-record backup — adding the character hints and the "translate from both" rule moved:

  • MRR from 0.803 to 0.809;
  • the worst-case configuration from 0.770 to 0.787;
  • queries with the target page in the top ten from 53 to 54.

The average gain is modest, as you would expect from a change that only affects Han-only queries. The value lies in the cases it rescues, and in removing a dependency: cross-lingual search for Han text no longer depends on whether a detector model happens to be installed.

Scope and limits

  • The tables cover everyday characters (level 1 of each standard) and Simplified Chinese. Traditional Chinese shares more characters with Japanese and follows the detector or "translate from both" path.
  • Mixed queries are decided by counts, which favors whichever language contributes more distinguishing characters.
  • This is a search heuristic, not a general-purpose language identifier. It only needs to choose a good translation direction, and when it cannot, it covers both.

The takeaway

When a statistical tool struggles with your hardest inputs, check whether the data already contains a deterministic signal. Here it did: national character-set standards record, quite precisely, which characters each language uses. Two set differences, computed by a forty-line script, made Han-only queries work with or without a detector.

And when the signal is truly absent, searching both readings can beat guessing one.

SpotRecall is open source under AGPL-3.0. spotrecall.com · source on GitHub · install from the Chrome Web Store.

This is Post 12 in Building SpotRecall, and the second post in Season 3: sharper, measured search.