Short answer: give itinerary retrieval a 300 ms p95 contract, spend most of it on one high-recall pass, and reserve a small, explicit slice for reranking and context assembly. A travel planner feels slow when every candidate gets an expensive semantic lookup, so the architecture should narrow the set before it asks for richer signals.
How should a travel itinerary planner divide retrieval latency budgets?
Start with a deadline that the product can defend. For a B2B SaaS planner that watches travel policy pages and turns them into trip options, I use 300 ms p95 for retrieval, separate from model generation. That is a budget, not a benchmark claim. Your mileage may vary when the source corpus is remote or freshly indexed.
The request carries a destination, dates, traveler constraints, and a change token from the last itinerary. A cheap lexical filter removes stale or impossible documents. A vector pass then retrieves a few dozen candidates. A reranker scores that short list, and a context builder clips passages to the model's input limit. Each stage returns its elapsed time and candidate count; the caller can make a deliberate fallback when a slice is gone.
Here is a small Node.js skeleton. It uses generic HTTP boundaries so the same contract works with a self-hosted index or a managed service.
type Budget = { deadlineMs: number; spentMs: number }; type Candidate = { id: string; text: string; score?: number }; const elapsed = (start: number) => performance.now() - start; async function retrieveItinerary(query: string, budget: Budget): Promise<Candidate[]> { const lexicalStart = performance.now(); const lexical = await lexicalSearch(query, 60); budget.spentMs += elapsed(lexicalStart); if (budget.spentMs >= budget.deadlineMs) return lexical.slice(0, 8); const vectorStart = performance.now(); const vector = await vectorSearch(query, 40); budget.spentMs += elapsed(vectorStart); const merged = mergeById([...lexical, ...vector]).slice(0, 40); if (budget.spentMs >= budget.deadlineMs) return merged.slice(0, 12); const rerankStart = performance.now(); const ranked = await rerank(query, merged.slice(0, 20)); budget.spentMs += elapsed(rerankStart); return ranked.slice(0, 8); } The important detail is the early return. It is an intentional quality trade: eight lexical matches are better than a timeout that leaves the planner with no answer. I log the branch, because a rising fallback rate points to a capacity or corpus problem rather than a prompt problem.
What fails when every retrieval stage gets the same deadline?
Uniform timeouts hide queueing. A vector request may be fast in isolation and slow under concurrent re-indexing; a reranker may have a stable median but a wide tail. If both receive 100 ms, the tail of the first consumes the second's share. Propagate one absolute deadline instead: deadline = requestStart + 300 and pass the remaining time to every dependency. For example, a request can leave the lexical stage with 210 ms remaining, spend 160 ms waiting for a busy vector shard, and arrive at reranking with only 50 ms. A per-stage timeout would still grant the reranker another 100 ms, quietly violating the user-visible contract. An absolute deadline makes the violation visible and lets the code return the already merged candidates instead of starting work it cannot finish.
Measure tails.
I also cap fan-out. Asking six regional indexes for twenty hits each creates 120 records to merge, deduplicate, and rerank. That work is CPU and memory pressure inside the API process, even when network calls are parallel. In practice, I start with two shards, 60 lexical hits, 40 vector hits, then measure recall before increasing either number.
Freshness creates another trap for a page-change product. An index that is ten minutes behind can return a beautifully relevant old policy. Store the source revision and fetched-at timestamp beside each chunk, and apply a freshness rule before ranking. If the rule rejects every candidate, report an empty result with a reason code; do not silently substitute unrelated travel advice.
A practical contract for quality, cost, and observability
Latency is only useful when paired with retrieval quality. Track recall@k on a small, labeled set of destinations, dates, and policy questions. Track p50 and p95 per stage, timeout fallbacks, duplicate rate after merging, and the percentage of context tokens that survive clipping. A single end-to-end number cannot tell you whether the lexical filter is too strict or the reranker is simply late.
| Decision | Helps | Costs or risk |
|---|---|---|
| Lexical pre-filter | Fast exact names, codes, and dates | Misses paraphrases |
| Vector retrieval | Better semantic recall | Index and embedding work |
| Reranking top 20 | Higher ordering precision | Tail latency and compute |
| Freshness gate | Prevents obsolete policy text | Can produce fewer matches |
Keep the instrumentation boring and structured. Emit one span per stage with candidate_count, elapsed_ms, deadline_remaining_ms, and fallback_reason. Sample full documents only in a protected debug stream; itinerary queries can contain passport or customer data. Redact query values in ordinary logs.
When is a simpler search path the right choice?
The catch is that a three-stage design is not suitable when the corpus is tiny, static, and already keyed by destination or booking code. In that case, a single indexed query is easier to operate and may leave more time for generation. Stick with the simpler path when its measured recall meets the product threshold and its p95 stays inside the same contract during an index rebuild.
Conversely, do not add a reranker just because semantic search is fashionable. If labeled tests show no meaningful gain for policy-change alerts, spend that slice on better chunk boundaries, source revision checks, or a warm connection pool. I am not sure which threshold will hold for your corpus; the experiment that resolves it is a replay of real itinerary queries with old and new page revisions, measured at the same concurrency as production.
The operational checklist is short: define the absolute deadline, bound fan-out, record candidate provenance, test stale revisions, and rehearse the fallback branch. Then load-test re-indexing and normal traffic together. A contract that survives that test is more valuable than a faster median on an idle laptop.