How SpotRecall measures every ranking change on real browsing data — with an offline replay that produces exactly the ranking the extension ships.
The previous three posts described what changed in SpotRecall 1.3.1's search: a steeper rank-fusion curve, script-based language detection for Han-only queries, and a set of small precision signals. This post is about how those changes were chosen — and why I trust the numbers.
SpotRecall is a private, on-device browser recall extension. That creates an unusual evaluation problem. The two most important inputs to its ranking — Chrome's on-device Translator and the extension's own vector store — exist only inside a running browser. You cannot import them into a unit test. Yet tuning a ranking means trying many variations, and a full browser round-trip per variation is far too slow.
The solution was to split evaluation into two steps: capture once in the real browser, then replay offline as many times as needed.
Step one: capture in real Chrome
A capture script drives the built extension in real Chrome, using puppeteer-core and a dedicated profile kept inside the eval folder. It:
- imports a SpotRecall backup (here, 1,088 records of my own browsing history);
- waits for the embedding model and downloads the translation models (Chrome only downloads them after a user gesture, which the script simulates with a click);
- for every labeled query, records the translation in every English/Japanese/Chinese direction, the vector-search results for every resulting text, and the live ranking the extension actually shows.
Recording every direction, not only the ones production uses, is deliberate. It lets the offline step evaluate other language-detection strategies without another browser run. The script translates from Chinese and from Japanese even when production would pick only one, so the replay can compare "pick one" against "translate from both."
Step two: replay offline with production code
The replay runs in Node, in seconds. Crucially, it does not reimplement the ranking. It imports the same modules the extension ships: language detection, the BM25 index, fusion, record priors, and deduplication.
That required one architectural choice early on: every piece of ranking logic is a pure function with its parameters passed in, and the extension's service worker only does the I/O around it. The fusion parameters live in one object that both the extension and the eval import.
With that in place, the replay supports several modes:
npm run eval:fusion -- --backup … # production vs baseline, per tag npm run eval:fusion -- --backup … --ablation # each improvement alone and stacked npm run eval:fusion -- --backup … --cv # 2-fold: tune on half, test on the other npm run eval:fusion -- --backup … --grid # parameter sweep npm run eval:fusion -- --backup … --bookmarked-backfill # worst case for title-only records Parity as a design goal
An offline replay is only useful if it reproduces the real thing. So the replay reports parity: for each query, does the offline top ten match the live top ten captured from the browser? The target is top-10 identical: n/n.
Reaching it took two details worth sharing.
Fresh service-worker code on every run. A reused Chrome profile keeps a cached copy of the extension's previous service-worker script. Extension pages load the new build, but the background worker can keep running the old one. The capture script now clears the profile's service-worker cache before each launch. The records live in IndexedDB and are untouched; the extension simply registers its worker again.
Identical tokens in Node and Chrome. SpotRecall tokenizes text with Intl.Segmenter, which handles Japanese and Chinese word boundaries. But Node and Chrome ship different ICU builds, and they segment punctuation inside words differently — Amazon.co.jp is one segment in one engine and three in the other. 1.3.1 re-splits every word-like segment on punctuation and symbols, so both engines produce exactly the same tokens.
That second fix pays off beyond evaluation. The extension stores a snapshot of its keyword index, and with ICU-independent tokens that snapshot stays consistent across Chrome updates. Because the token format changed, the release bumps the snapshot key and clears the old snapshot automatically to reclaim its space.
With both in place, every number in this series was measured on a replay that matches the live extension query for query.
The acceptance rule
A few dozen labeled queries is a realistic budget for a personal project. It is also small enough to overfit easily. Each query in the set carries a tag: xling (cross-lingual), same (same language), kanji (Han-only Japanese), or bookmark (title-only targets). A change was accepted only if it held up on three checks at once:
- per tag, so a gain on one kind of query does not hide a loss on another;
- in the worst case, a simulation where every backfilled bookmark also receives the bookmark boost, as it would in a bookmark-heavy profile;
- under 2-fold cross-validation: tune on half of the queries, measure on the other half, then swap.
The third check made the most visible difference. The grid search's best parameters scored well on the queries they were tuned on. On held-out queries, they did not beat the shipped values. So the shipped values are the ones that generalized, not the ones that topped the grid.
Ablation: each improvement, stacked
Measured on 54 mixed Chinese, Japanese, and English queries, every number live-identical:
- Baseline (k=3, translations ×0.7, trimmed vector lists): MRR 0.803, Hit@1 38, Hit@10 53, worst-case MRR 0.770.
- + Han character hints, ambiguous Han translated from both zh and ja: MRR 0.809, Hit@1 38, Hit@10 54, worst case 0.787.
- + title-only records ×0.5 unless they match every term of a variant: MRR 0.828, Hit@1 39, Hit@10 54, worst case 0.826.
The average gains are steady rather than dramatic. The worst-case column shows the larger effect: the configuration most exposed to title-only noise improved the most.
Ideas that did not meet the bar
The rule is only meaningful if it sometimes says no. Two promising ideas were tested and set aside:
- Dropping the keyword lists of low-coverage translations. It helped Han-only Japanese queries, made cross-lingual ones worse, and was unstable across the two folds.
- A recency prior. Favoring recently visited pages sounds natural for a recall tool, but it pushed down old bookmarks that users keep precisely because they matter.
Both remain easy to revisit. The harness makes trying them again a single command.
Working with an AI pair
As with the rest of this series, the work was done with an AI pair. In this release, that meant the capture script, the replay, and the ablation and cross-validation modes could be built and rerun quickly — a full parameter sweep takes seconds.
Speed is the easy part. What makes it trustworthy is a fixed acceptance rule and a parity check that does not depend on anyone's judgment in the moment. When a cheap iteration loop proposes a better-looking number, the rule decides whether it ships.
Privacy, by construction
The labels for this evaluation describe my own browsing, so they are treated like the data they are. The labeled-query file and all backups are git-ignored and never enter the repository. The repository contains only an example labels file that shows the format:
[{ "query": "…", "match": ["url:<regex>", "title:<regex>"], "tag": "xling" }] Anyone can run the same loop on their own SpotRecall backup, locally.
Scope and limits
Fifty-four queries on one person's 1,088-page history is a methodology demonstration, not a benchmark. The trends are what matter: which changes hold up across query types, in the worst case, and on held-out queries.
The takeaway
When part of your system only runs in a browser, capture its outputs once and replay everything else with production code. Make the replay prove parity with the live system. Then tune against a rule that includes held-out data — and let that rule, rather than the best-looking cell of a grid, decide what ships.
SpotRecall is open source under AGPL-3.0. spotrecall.com · source on GitHub · install from the Chrome Web Store.
This is Post 14 in Building SpotRecall, and the final post in Season 3: sharper, measured search.
