The retrieval architecture should be chosen by the failures your evaluation set can expose, not by a single recall score. For a restaurant menu assistant, I would version a compact evidence corpus, measure top-k recall and citation validity, and enforce a latency budget. The same discipline protects a logistics page watcher: a missed change and a late alert are different incidents and need different signals.
Short answer: build labeled evaluation sets around menu intent, freshness, and refusal cases; compare lexical, dense, and hybrid retrieval against those sets at production concurrency; then keep the path whose evidence quality fits the alert deadline.
The missed-alert lesson
I run cron and queue infrastructure in production, and I have been paged for both missed jobs and duplicate deliveries. A page watcher that fetches a carrier's service page has the same operational shape as a menu assistant: fetch a revision, normalize it, retrieve the relevant evidence, and publish a notification or answer. Treating those as one “search quality” number makes the postmortem harder. During review, I want to see the exact input revision, the normalized hash, the queue attempt ID, the selected chunks, and the deadline at each hop. Without that chain, an engineer can prove that a request returned quickly while still missing the changed paragraph, or prove that the right paragraph existed in the index after the alert had already been dropped. The test set has to preserve those distinctions so a replay can identify the first wrong state instead of assigning blame to whichever component logged last.
The page was unchanged.
The invariant I want is simple: every returned chunk has an owner, a source revision, and a reason it is acceptable evidence for this query. If a parser changes whitespace, the watcher should not page the team for a cosmetic diff. If a restaurant removes an allergen statement, the assistant should not answer from yesterday's chunk. Those are evidence-boundary failures, not ranking trivia.
Start with rows in source control. Each row needs the user question, restaurant and location, source revision, accepted evidence IDs, and an expected action: answer, ask a clarification, or abstain. Include hard rows: two dishes with the same name, a branch override, a seasonal price, an item deleted in the newest revision, and OCR that drops a “contains” marker. Keep the original source span beside the label so a reviewer can inspect why a hit is valid.
One rule saves time: prove the miss.
What should a retrieval architecture evaluation set measure for a menu assistant?
Measure several slices instead of publishing one blended score. I use recall@5 to ask whether any acceptable chunk appears, precision of the restaurant and location filter, revision correctness, and citation-span validity. Record p50 and p95 retrieval latency at the same concurrency and deadline as the user path. A result that is relevant after the request has timed out is operationally equivalent to no result.
| Evaluation slice | Failure it reveals | Useful pass signal |
|---|---|---|
| Exact item and alias | tokenization, spelling, menu synonyms | accepted item in top 5 |
| Attribute question | allergen, calorie, modifier mismatch | span contains the requested attribute |
| Time and location | branch hours or local override leakage | current scoped revision wins |
| Change replay | stale additions, edits, and deletions | newest revision or explicit abstention |
| Ambiguous intent | two dishes or locations with one name | clarification instead of a guess |
| Unsupported claim | model fills gaps from prior knowledge | refusal with no fabricated citation |
Keep a small smoke set for every pull request and a larger canary set for index or embedding changes. A practical starting point is 12 carefully chosen smoke rows and 60–100 canary rows per restaurant class; tune that size to the diversity of your menus rather than treating it as a universal benchmark. I'm not sure one global threshold will survive every cuisine and publishing workflow. Your mileage may vary, so keep thresholds per slice and review false negatives with the source document open.
Building a bounded, revision-aware retrieval path
Separate ingestion, retrieval, generation, and notification. Ingestion assigns a stable document ID, revision, chunk ID, and source locator. Retrieval receives structured restaurant_id, location_id, and revision filters; do not hide those constraints in prose appended to a query. Generation sees only validated hits. The application records query ID, hit IDs, revision, elapsed milliseconds, and the answer or alert decision before publication.
For the logistics watcher, persist the fetched page's ETag or Last-Modified value when available, and hash the normalized content. HTTP validators let the fetcher distinguish “unchanged” from “changed but equivalent after normalization”; the same revision field then flows into the evaluation replay. A deleted menu item follows the same lifecycle: write the new desired chunk set, remove IDs absent from that set, mark reconciliation complete, and only then serve the revision.
The synchronous path needs a hard deadline. A hybrid first stage can combine lexical matching for dish codes and negations with dense matching for paraphrases; a small reranker may improve ordering, but it must have a timeout and a deterministic fallback. Cache immutable revision IDs, never a vague key such as today-menu.
Here is the contract I keep above any search implementation. It is intentionally small: the evaluator owns the assertion, while an adapter owns filtering and ranking details.
package eval import ( "context" "fmt" ) type Hit struct { ID string Restaurant string Location string Revision string Text string Score float64 } type Retriever interface { Search(ctx context.Context, query, restaurant, location, revision string, limit int) ([]Hit, error) } func ContainsEvidence(hits []Hit, accepted map[string]bool) bool { for _, hit := range hits { if accepted[hit.ID] { return true } } return false } func Evaluate(ctx context.Context, r Retriever, query, restaurant, location, revision string, accepted map[string]bool) error { hits, err := r.Search(ctx, query, restaurant, location, revision, 5) if err != nil { return fmt.Errorf("retrieve %q: %w", query, err) } if !ContainsEvidence(hits, accepted) { return fmt.Errorf("no accepted evidence for %q", query) } return nil } The production adapter should reject a hit whose restaurant, location, or revision does not match the request, even if its similarity score is high. That check catches cross-branch leakage before a language model can turn it into a confident sentence. It also makes a replay useful: the same query and revision can be run against an index snapshot and compared by hit ID, not by an unstable score alone.
Choosing recall without surrendering the deadline
Lexical retrieval is usually strong on exact item names, ingredient tokens, and “without” negations. Dense retrieval helps with paraphrases such as “no dairy” versus “milk-free,” but can blur two similarly described dishes. Hybrid retrieval plus a bounded reranker is a reasonable test candidate, not a default answer.
The trade-off is visible in the evaluation table: increasing top-k gives the reranker more evidence but consumes memory and time; adding a second model can lift recall while making p95 unpredictable. That path is not suitable for a voice interaction with a strict sub-second budget or for a watcher that must alert before the next dispatch window. Stick with lexical filtering and an explicit “I need to check the current menu” response when the deadline is tighter than the measured reranker path.
Cost belongs in the same worksheet as recall, freshness, and operator effort. Count index storage, embedding work on revisions, reranker calls, queue retries, and the time needed to inspect a failed row. Do not make a price claim the decision rule; a cheap index that creates duplicate alerts still spends the on-call budget.
Operating the set after launch
Run the smoke set on every code change and the canary set on every index rebuild. When a row fails, print the old and new top-five IDs, revision metadata, the missing evidence span, and the latency percentile that applied. Keep one representative failure in the runbook with its source input and expected action.
Schedule a deletion replay weekly. Remove an item from a fixture, ingest the replacement revision, and verify that the assistant finds the replacement or abstains. For the page watcher, replay an unchanged fetch, a semantic change, and a formatting-only change. Route evidence misses to the search owner and latency breaches to the platform queue; separate pages make the response faster.
This approach does not guarantee that a menu is correct when the restaurant publishes contradictory source files. It is not suitable when labels cannot be reviewed or when policy requires a human to approve every allergen answer; add that review gate and measure its queue delay. The point is to make the boundary explicit, so a new retrieval component can be compared without rewriting the incident playbook.