Retrieval Architecture: Four Evaluation Sets for News Monitoring Latency

Short answer: build four small, labeled evaluation sets before choosing an index. For a fintech news monitoring service, the winning architecture is the one that keeps high-risk stories in the top results without breaking the latency budget. A bigger benchmark is not automatically a better one.

Start with a decision table

Treat those rows as building blocks, not product categories. An inverted index such as Elasticsearch is a sensible sparse baseline. A vector engine such as Vespa can support dense and hybrid flows. PostgreSQL with pgvector is often a practical fit when relational joins and a modest corpus dominate. Each can be the wrong choice: a dense-only path can miss a ticker, while a large distributed index can add operational latency that the news desk will feel.

What should a news monitoring retrieval architecture measure first?

The query is an evaluation contract. Write down the event that should trigger an alert, the evidence an analyst needs, and the maximum age of an acceptable result. Then create four sets, each with a different failure mode. I initially thought one random sample would be enough; it wasn't. Random rows overrepresent routine headlines and hide the rare, high-cost query that wakes an analyst at 06:00.

1. Entity set. Tickers, issuer names, subsidiaries, people, and spelling variants. Include punctuation and aliases.
2. Event set. Queries such as “bank raises CET1 target” or “sanctions inquiry opened.” Label the relevant article and the relevant passage, not just the document.
3. Temporal set. The same event across a wire update, a correction, and a later analysis. Relevance without freshness is a false win.
4. Adversarial set. Near matches, syndicated copies, boilerplate risk language, and negations such as “no breach found.”

Keep a frozen test slice and a changing production slice. The frozen slice catches regressions; the changing slice catches vocabulary drift. I like a plain CSV with query_id, document_id, relevance, event_time, and label_reason. The reason field sounds fussy until an analyst asks why an article was included at 03:00. Keep annotator guidelines beside the data, version them, and record disagreements instead of silently averaging them. A label that changed from “related” to “actionable” is a data migration, not noise.

A two-stage design that makes the trade-off visible

Picture a funnel. The first stage gathers 100 candidates quickly. The second stage reads their titles, snippets, timestamps, and source trust metadata, then returns 10. This separation lets you tune recall and precision independently.

Measure the tail.

Chunking is part of retrieval quality. News articles often put the decisive number in a table or a paragraph far below the lead. Store a stable document id, section offsets, publication time, and a content hash with every chunk. Use overlap only where sentences cross boundaries; excessive overlap inflates duplicate hits and reranker work.

Here is a minimal TypeScript harness for comparing runs. It keeps latency beside quality, so a gain in NDCG can't hide a slow tail.

type Hit = { documentId: string; score: number; latencyMs: number }; type Judgement = { queryId: string; documentId: string; relevance: number }; function ndcgAt(hits: Hit[], labels: Judgement[], k: number): number { const grade = new Map(labels.map((x) => [x.documentId, x.relevance])); const dcg = hits.slice(0, k).reduce((sum, hit, i) => { const gain = (2 ** (grade.get(hit.documentId) ?? 0)) - 1; return sum + gain / Math.log2(i + 2); }, 0); const ideal = [...labels].sort((a, b) => b.relevance - a.relevance).slice(0, k); const idcg = ideal.reduce((sum, item, i) => sum + ((2 ** item.relevance) - 1) / Math.log2(i + 2), 0); return idcg === 0 ? 0 : dcg / idcg; } function p95(values: number[]): number { const sorted = [...values].sort((a, b) => a - b); return sorted[Math.ceil(sorted.length * 0.95) - 1] ?? 0; } 

Log the set name, index version, chunking configuration, query class, result count, and p50/p95 latency for every run. Export traces with a correlation id. Metrics tell you that p95 moved; a trace tells you whether tokenization, the vector lookup, or reranking moved it. In one useful drill, inject a deliberately slow reranker and verify that the trace points to that span while the quality score stays unchanged. A 429 from a dependency should be counted separately from a genuine no-result query, or the alert rate becomes meaningless. Alerts should fire on quality regressions in the frozen set and on latency regressions in production, with separate thresholds.

Pick a rollout rule, then keep the limits

Ship a change only when entity recall does not fall, event-set NDCG improves or holds, and p95 stays inside the service objective. Review ten false positives and ten false negatives manually after each index or embedding change. That small ritual catches label mistakes faster than another dashboard.

The catch is scope. Evaluation sets cannot prove coverage for a brand-new crisis, a source that changes its markup, or a language absent from your labels. A hybrid design is also not suitable when the team cannot operate two retrieval paths; stick with a well-instrumented sparse baseline until ownership is clear. Your mileage may vary on the exact k and latency target, because alert urgency differs by desk.

Do not optimize for a single leaderboard number. In monitoring, a missed filing and a duplicate headline have different costs. Make those costs explicit in the labels, expose them in traces, and let the architecture follow the evidence.

References

• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
• https://docs.vespa.ai/en/
• https://github.com/pgvector/pgvector
• https://opentelemetry.io/docs/