Short answer: build a small, versioned evaluation set from real documentation questions, score retrieval with recall@k and ranking quality, then choose the fastest index that clears your relevance threshold. For a logistics team watching a carrier web page, keep latency as a constraint, not as a substitute for finding the paragraph that explains the change.
| Retrieval approach | Pick it when | Watch for |
|---|---|---|
| Lexical search (BM25) | Error codes, route names, and exact terms dominate | Synonyms and paraphrases can be missed |
| Vector search | Engineers ask concept-shaped questions across varied wording | Similar-sounding passages can outrank the answer |
| Hybrid search | Queries mix identifiers with natural language | Two scores need a repeatable fusion rule |
The evaluation set is the control surface. Without it, a new embedding model or chunk size is only a feeling in a dashboard. Store the query, the expected document or passage, and the acceptable rank in a file that can be reviewed like code.
Measure it twice.
What should a semantic retrieval evaluation set contain for internal engineering documentation?
Start with the question types your on-call team actually asks. A logistics example might be: “What changed in the carrier cutoff page after the 02:00 crawl?” The answer is a dated diff record, while the supporting context may be the parser runbook and the alert policy. Label both the primary passage and acceptable secondary passages; otherwise a useful runbook hit looks like a false negative.
I initially treated every query as a single-answer lookup. That made the score look precise and the review painful. Some questions have several valid passages, and a page-change alert often needs the newest version, not merely a semantically similar older one.
Use five small slices: exact identifier, paraphrase, multi-hop procedure, negative query, and freshness-sensitive query. Give each slice its own count. A 50-query set with 10 items per slice is easier to inspect than one opaque score, and it exposes a regression such as “good prose, bad error-code matching.” Keep a stable query_id; append new cases instead of rewriting old labels. For each case, retain the source URL, repository path, revision, and the human explanation for why a passage is acceptable. That extra context turns a score review into an engineering conversation: the person changing chunking can see whether a miss came from tokenization, metadata filtering, or an outdated label, then update one fixture and rerun the same slice instead of arguing from a dashboard aggregate.
Here is the kind of failure worth preserving. Query carrier-cutoff-02 retrieves the current alert policy at rank 1, an old carrier page at rank 2, and the parser runbook at rank 3. A naive evaluator marks that as a pass because the right topic appeared. A production reviewer marks it as a freshness failure because the alert must cite the current page revision. Keep the case, add the revision constraint to its label, and make the next index run prove that the old page no longer wins. That single example tests chunk metadata, ranking, and the alert contract together; it is more informative than another dozen easy paraphrases.
A negative query matters. “Which carrier page is covered by this alert?” should return no confident match when that carrier is outside the monitored set. Treating the top vector as an answer creates a plausible citation for an absent fact.
How do you measure retrieval quality and latency without fooling yourself?
For each query, record the ranked IDs returned by the retriever and a timestamp for the request. Recall@k asks whether any acceptable passage appears in the first k results. Mean reciprocal rank rewards putting the first acceptable passage near the top. Neither metric says the text is current, so add a freshness check for documents with revisions or crawl timestamps.
Here is a compact TypeScript scorer. It assumes labels are passage IDs, not generated answers. That distinction keeps retrieval evaluation separate from generation quality.
type Case = { queryId: string; relevant: Set<string>; expectedMaxAgeHours?: number; }; type Result = { id: string; ageHours?: number }; export function score(cases: Case[], ranked: Map<string, Result[]>, k: number) { let recalled = 0; let reciprocalRank = 0; for (const item of cases) { const results = ranked.get(item.queryId) ?? []; const top = results.slice(0, k); const hit = top.findIndex((r) => item.relevant.has(r.id)); if (hit >= 0) { recalled += 1; reciprocalRank += 1 / (hit + 1); } } return { recallAtK: cases.length === 0 ? 0 : recalled / cases.length, meanReciprocalRank: cases.length === 0 ? 0 : reciprocalRank / cases.length, }; } Run the same cases against lexical, vector, and hybrid candidates. Capture p50 and p95 latency beside the quality scores. A p95 spike is operationally different from a small mean change: page-change alerts arrive in bursts, so tail latency can delay a notification even when average retrieval looks fine.
The practical gate is two-dimensional. Set a minimum recall for each slice, then set a p95 budget for the alert path. If hybrid search raises recall for error-code queries but breaks the budget, route only those queries to hybrid or reduce the candidate set. Your mileage may vary; the right threshold depends on how costly a missed change is.
Which indexing and chunking choices survive real documentation churn?
Chunk by meaning before tuning vector dimensions. Keep a heading with the procedure it governs, preserve code blocks as intact units, and attach metadata such as repository, path, revision, and crawl time. For a changed carrier page, store the old and new fingerprints so the evaluator can ask for the current passage explicitly.
Elasticsearch supports BM25 and vector fields in one search system; PostgreSQL with pgvector keeps embeddings beside relational metadata; Vespa offers ranking expressions for combining signals. These are different operational shapes, not a leaderboard. Pick the system your team can back up, inspect, and reindex predictably. A hybrid design still needs one canonical document ID so score comparisons remain meaningful.
The failure mode I watch is stale truth. A semantically perfect match to last month’s cutoff is wrong for today’s alert. Add a freshness feature or a hard filter, and put at least one intentionally stale case in every evaluation run.
When a case fails, save the query, top five IDs, scores, document revisions, and latency. A reviewer should be able to explain the failure without rerunning production traffic. Group failures by cause: bad label, missing chunk, synonym gap, stale metadata, or ranking tie.
Keep a tiny change log next to the set. Record why a label changed and which parser or index version caused it. A 202 response from an indexing job is not evidence that the passage is searchable; verify visibility in the same environment used by the alert worker.
One short rule: never optimize an aggregate while a slice is red.
Limits and trade-offs
Evaluation sets do not remove judgment. Labels can encode one team’s vocabulary, and a five-slice set can underrepresent rare incidents. Expand it from reviewed misses, but freeze a regression subset so improvements cannot erase old failures.
Vector retrieval is not suitable when exact identifiers are the whole query; stick with lexical or hybrid search then. A large hybrid stack is also a poor fit for a small documentation corpus that changes weekly, where a simpler index is easier to reindex and audit. The catch is maintenance: every new chunking rule, embedding model, and freshness policy changes the behavior you must measure.
