TL;DR — Running vector search at warehouse scale forces you to partition the index by tenant, time, or shard — but that partitioning breaks the global top-k guarantee vector search is supposed to provide. You end up stacking two approximations: the ANN algorithm's recall loss, and a merge-across-shards ranking error that almost nobody measures. The fix isn't a better index, it's treating top-k as a distributed query problem with its own correctness budget.
Every vector database pitch includes a promise: give me a query embedding, I'll give you the k nearest neighbors. That promise is already approximate — HNSW, IVF, and their cousins trade exact nearest-neighbor search for speed, and everyone building on them accepts that trade. What almost nobody accounts for is the second approximation that shows up the moment you scale past a single index: partitioning.
At warehouse scale — hundreds of millions to billions of vectors, multi-tenant, continuously ingesting — you cannot keep one monolithic index. You shard it. And sharding doesn't just change where data lives. It changes what "top-k" means, in ways that degrade retrieval quality silently, long after your latency dashboards look fine.
Top-K Is a Global Claim, Not a Local One
When someone asks for the top 10 nearest vectors to a query, the implicit claim is global: out of everything in the collection, these are the closest 10. A single HNSW graph over the full dataset approximates that global claim reasonably well — recall loss comes from the graph's traversal heuristics, not from missing data.
The moment you split that graph into shards — by tenant, by time window, by geography, by hash — you've changed the query from "find the 10 closest vectors in the dataset" to "find the 10 closest vectors in each shard, then merge." Those are not the same operation, and the gap between them grows with shard count and with how skewed your data distribution is across shards.
Why Warehouse Scale Forces the Split
This isn't optional at scale. A few forces push every vector database toward partitioning once you're past the single-node, single-tenant demo:
Index build cost. HNSW graphs and IVF clusters are expensive to build and even more expensive to rebuild. Past a certain size, you shard so that ingestion and compaction stay tractable instead of becoming a full-collection rebuild every time.
Tenant isolation. Multi-tenant RAG systems shard by tenant for both performance and data-governance reasons — one noisy tenant shouldn't degrade another's query latency, and one tenant's vectors shouldn't leak into another's results through a shared index.
Hot/cold separation. Time-partitioned data — logs, chat history, event embeddings — gets sharded by time so recent data stays in fast, memory-resident indexes while older shards move to cheaper storage tiers.
Replica and placement constraints. Distributed deployments shard for the same reasons distributed databases always shard: no single node can hold everything in memory, and you want parallelism across the fleet.
None of these are mistakes. They're the correct engineering response to scale. But each one turns a single approximate search into a fan-out-and-merge problem, and the merge step is where correctness quietly leaks out.
The Merge Problem Nobody Benchmarks
Here's the mechanic. You fan a query out to n shards, each shard's index returns its local top-k using its own approximate search, and a coordinator merges those n lists by score to produce the final top-k. This works correctly only if two conditions hold: every shard's local top-k actually contains the true global top-k candidates from that shard, and scores are comparable across shards.
Both conditions fail in practice, in ways that compound rather than cancel.
First, local top-k from an ANN index is already a lossy sample of that shard's true top-k. If you ask each shard for its local top 10 and merge, you're assuming the true global top 10 is distributed such that no shard needed to contribute more than 10 strong candidates. At warehouse scale with skewed data — one tenant's shard is denser in the query's semantic neighborhood than others — that assumption breaks. The shard holding the actual best matches might return a locally-correct top 10 that excludes some of the globally-best vectors simply because the local index's graph traversal didn't surface them given the query's entry point.
Second, score comparability across shards is not guaranteed unless every shard uses identical index parameters, identical distance metrics, and ideally was built from the same embedding model version. In practice, shards get rebuilt at different times, with different index configs, sometimes even different quantization settings for storage efficiency. A cosine similarity score of 0.82 from a freshly rebuilt HNSW shard and a 0.82 from an older, more heavily quantized shard are not the same confidence of match — but the merge step treats them as if they were.
Where This Actually Bites
This failure mode doesn't show up as an outage. It shows up as a slow, hard-to-diagnose quality regression. A RAG system's retrieval quality drifts downward as the underlying collection grows and gets re-sharded, even though the embedding model hasn't changed and the index build logs show no errors. Support tickets say "the assistant used to find the right document, now it sometimes doesn't," and the on-call engineer checks latency, checks index health, checks the embedding pipeline — and finds nothing wrong, because nothing in the usual observability surface captures merge-stage ranking error.
It's worse in filtered search, which is the norm in production RAG rather than the exception. A query that's scoped to a tenant plus a metadata filter plus a time range effectively narrows the candidate shards further, sometimes down to a single shard with a thin population of matching vectors. The local top-k from that thin shard can be a poor approximation of relevance, but the system has no signal that it degraded — the query still returns exactly k results with plausible-looking scores.
What Actually Helps
You can't eliminate the shard-and-merge tradeoff at warehouse scale, but you can budget for it instead of ignoring it.
Over-fetch per shard. Ask each shard for more than k results — commonly a multiple of k — before merging, so local recall loss has room to be corrected at the merge stage rather than baked in before the coordinator ever sees the candidates.
Normalize scores at build time, not query time. Pin distance metric, quantization scheme, and embedding model version per shard, and track those as metadata the coordinator can check before trusting a cross-shard score comparison.
Measure recall against brute force, periodically, on a sample. Latency and error-rate dashboards won't catch this. A scheduled job that compares sharded top-k against an exact brute-force search over a sampled subset is the only reliable early-warning system for merge-stage degradation.
Treat shard rebalancing as a correctness event, not just a capacity event. When you re-shard for storage or tenancy reasons, re-run the recall benchmark. A rebalance that improves write throughput can silently worsen retrieval quality if it changes how semantically similar vectors get distributed across shards.
Vector search was sold as a drop-in replacement for exact nearest-neighbor lookup, with the understanding that you'd trade a little recall for a lot of speed. That trade is fine in isolation. The problem is that warehouse-scale deployments don't make that trade once — they make it twice, once inside the index and once across the fleet of shards that index gets split into, and only the first one is ever in the benchmark deck.
