GraphRAG Is an Inference Problem, Not a Database Problem

Vector RAG embeds chunks of text, retrieves the nearest neighbors to a query by cosine or dot-product similarity, and puts them in context. It answers “what does this document say” and it’s cheap: one embedding call per chunk at index time, one embedding call per query.

GraphRAG replaces that single embedding pass with a multi-stage pipeline. At index time, an LLM reads each chunk and extracts entities and relationships as structured triples: “Company X, acquired, Company Y.” Microsoft’s original implementation then runs the Leiden algorithm, a community-detection method, on the resulting graph to group densely connected entities into clusters, and recurses the process to build a hierarchy: broad, generic communities at level 0, progressively narrower and more specific communities at each level above it. An LLM then writes a natural-language summary of every community at every level. That’s two full LLM passes over the corpus before a single query has been asked. At query time, the system pulls in the relevant community summaries (global search) or walks specific entity relationships (local search) instead of, or alongside, nearest-neighbor lookup.

That two-pass indexing step is what makes GraphRAG expensive, and it’s also what lets it answer a class of question vector RAG structurally cannot: “what does the whole corpus imply when you connect the entities across documents.” Vector RAG has no representation of that connection. It only has similarity between text and query.

GraphRAG-Bench, an ICLR 2026 benchmark, tests graph retrieval against plain chunk retrieval separately across four query types instead of reporting one blended score. The results:

Simple fact retrieval: chunks score 60.9% accuracy, graph scores 60.1%. A tie, and vector search arguably has a slight edge at pulling one fact from one document.

Multi-hop reasoning: graph wins, 70.3% versus 67.0%. Contextual summarization, questions that require synthesizing information spread across many documents: graph wins 64.4% versus 51.3%, a 13-point gap.

Aggregation queries: graph wins 53.4% versus 42.9%, a 10.5-point gap.

That’s the real shape of the result: a specialized tool for multi-hop and synthesis questions, not a universal upgrade. The much larger gaps quoted elsewhere, some papers report 50-plus point swings, come from a different and looser methodology: an LLM judge picking which of two answers it prefers on broad “make sense of the whole corpus” prompts, tested on one narrow dataset.

Microsoft’s original GraphRAG paper reported 72-83% win rates on comprehensiveness and 62-82% on diversity using that exact setup. The LightRAG paper reported an 83.6% win rate against naive RAG’s 16.4% on its legal-documents subset, a 67-point swing. Those are real numbers, but they measure which answer a judge model liked better on whole-corpus prompts over a single domain. They are not accuracy scores on general question answering, and they don’t transfer to a typical production workload. Check which methodology produced a number before you build a budget around it.

Use GraphRAG for: multi-hop questions, cross-document synthesis, and “how does everything in this corpus relate” queries, such as a startup mapping obligations across thousands of contracts, tracing a bug across a codebase and its related tickets, or connecting entities across years of research papers.

Don’t use it for: single-document fact lookup or FAQ-style retrieval, where the answer lives in one place and the job is just finding it. Vector RAG ties or beats graph retrieval there, at a fraction of the infrastructure cost and none of the two-pass indexing latency.

Before evaluating GraphRAG at all, beat the real baseline: hybrid retrieval combining BM25 keyword search, dense embeddings, and a cross-encoder reranker. Published benchmarks put the recall lift from adding hybrid fusion and reranking to plain vector search at roughly 15-30% relative improvement, with one widely cited comparison moving recall@10 from 78% to 91%. Most teams evaluating GraphRAG haven’t run this cheaper baseline first, and it closes a meaningful share of the synthesis gap before a graph enters the picture.

Index-time entity extraction makes up roughly 75% of total indexing cost in a full GraphRAG pipeline. The reason is structural: extraction runs one LLM call per chunk, chunks are small (typically 300 to 600 tokens versus the much larger windows an embedding call can process at once), and community summarization adds a second full LLM pass on top. Together that inflates total tokens processed by an estimated 3-5x compared to indexing the same corpus for vector RAG. At the scale of a real production corpus, that’s millions of LLM calls, not a database provisioning line item.

Query time carries its own tax. Global search has to read every community summary at the relevant level before synthesizing an answer; local search has to traverse multiple relationship hops and pull in connected entity descriptions. Both add LLM round-trips on top of retrieval. Published comparisons put full GraphRAG query latency at 2-3x plain vector RAG, end to end, before generation even starts.

Two consequences follow. First, the cost objection is dead. Indexing a 5GB legal corpus with Microsoft’s original GraphRAG cost roughly $33,000 in early 2024, a figure that ended nearly every internal GraphRAG proposal for two years. LazyGraphRAG, released by Microsoft Research in mid-2025, removes both expensive LLM passes from indexing entirely. It uses NLP noun-phrase extraction, not an LLM, to build a concept co-occurrence graph, then defers all LLM reasoning to query time using an iterative best-first and breadth-first search that only pays for relevance testing on the specific query asked.

That brought indexing cost down to parity with plain vector RAG, about 0.1% of the original full-GraphRAG cost, while matching GraphRAG’s global-search answer quality at more than 700x lower query cost. LightRAG closes a similar gap differently: it indexes with an LLM but retrieves through dual-level keys, low-level keys for specific entities and relationships, high-level keys for broader themes, and supports incremental graph updates so new documents merge into the existing graph without a full re-index. The $33,000 objection no longer applies to either system. What’s left is an infrastructure question: where do you run the extraction model, and how close is it to your graph store, vector store, and generation GPU?

Second, extraction doesn’t need a frontier model. Pulling entities and relationships out of a chunk and writing them into a fixed schema is a narrow, structured task, well suited to a small model running under vLLM with grammar-constrained structured-output decoding (JSON mode, enforced by a backend like xgrammar or outlines). Published vLLM benchmarks on Llama-3.2-3B report throughput of roughly 900-1,200 tokens per second on a single A100 at typical batch sizes, and larger open models on vLLM have cleared 15,000-plus tokens per second on one GPU at high concurrency. At that throughput and price point, extraction is a batch job on commodity GPU capacity, not a reason to route millions of chunks through a metered frontier API.

A GraphRAG deployment has four components that talk to each other constantly: the extraction model making index-time LLM calls, the graph store, the vector store handling the retrieval half of hybrid search, and the orchestration layer running traversal and generation. Every hop between them that crosses a network boundary, a different region, a different provider, storage that isn’t next to compute, adds latency. That latency compounds across the multiple query-time round-trips a graph traversal already requires, on top of a workload already running 2-3x slower than plain vector RAG.

Data location is a first-order latency variable here, the same way it is for any storage-to-GPU path: there is a measurable millisecond gap between a GPU reading data from the same data center, from a different data center, and from a different provider’s network entirely. Stack three or four of those hops into one graph traversal and the gap stops being trivial.

That argument favors running extraction, both stores, and generation in one region on one stack, rather than a managed graph database from one provider, a vector database from another, and inference wherever GPU capacity happened to be cheapest that week. Concretely, on DigitalOcean: run extraction as a batch job on Serverless Inference for bursty index builds, since it’s metered and scales down to zero between corpus updates. Move query-time extraction and generation to Dedicated Inference once traffic is steady enough to justify reserved GPU Droplet capacity (options span H100, H200, Blackwell B300, and AMD MI300X/MI350X/MI355X, on a 400G RoCE RDMA fabric across roughly 20 global data centers).

DigitalOcean’s inference stack runs a custom vLLM fork with tuned KV-cache management and speculative decoding, which matters for the query-time half of the pipeline where latency, not just throughput, is the constraint. DigitalOcean’s Vector Databases product supports pgvector on Managed PostgreSQL, alongside Weaviate and OpenSearch, covering the retrieval half of a hybrid pipeline in the same account and region as the graph store and the GPUs. See our companion piece on the cold-storage-vs-hot-inference latency hierarchy for this same argument made in general form.

Extraction layer: vLLM serving a small instruction-tuned model in the 3B to 8B parameter range, with structured-output decoding enforcing the entity/relationship schema, run as a batch job against GPU Droplets sized to the corpus. This is the step that accounts for roughly 75% of total pipeline cost, and it benefits the most from being self-hosted instead of metered per token against a frontier API.

Graph store: FalkorDB, Kuzu, or Neo4j, chosen by scale and query pattern, not brand recognition. FalkorDB represents the graph as sparse matrices and evaluates traversal as linear algebra, which keeps multi-hop and aggregate query latency predictable, one published benchmark reports first-query readiness in under half a millisecond. Kuzu is an embedded, disk-based engine suited to analytical batch workloads rather than high-concurrency serving. Neo4j trades some of that specialization for a larger ecosystem, more mature tooling, and an easier hiring pool if your team needs to scale support.

Vector store: Qdrant or pgvector for the hybrid-retrieval half of the pipeline. Use pgvector on Managed PostgreSQL if you want one fewer service to operate and your embedding volume is moderate. Move to Qdrant once you’re past the scale where that operational simplicity stops paying for itself, typically in the tens of millions of vectors.

Pipeline: LightRAG or LazyGraphRAG instead of the original full-GraphRAG implementation. Both now index at close to vector-RAG cost while keeping the multi-hop and synthesis accuracy gains documented above. Pick LazyGraphRAG for corpora that change constantly and where query volume is unpredictable; pick LightRAG where you want persistent, queryable entity and relationship data and can tolerate LLM cost at index time in exchange for cheaper, dual-level retrieval later.

Observability: instrument token spend per chunk at index time and latency per hop at query time separately, not as one aggregate number. A pipeline with two LLM passes at index time and two to three round-trips at query time will hide which stage is slow or expensive if you only track end-to-end numbers.

Placement: all four components in one region, on one provider, so the frequent index-time and query-time round-trips aren’t paying cross-network latency on every hop.

We’re publishing an open-source benchmark harness that runs four query classes, simple fact, multi-hop, aggregation, and contextual summarization, against three pipelines, hybrid vector RAG, LightRAG, and full GraphRAG, on identical DigitalOcean GPU Droplet hardware in one region. Same corpus, same extraction model, same generation model, only the retrieval architecture changes between runs. Index cost, re-index cost, P50/P99 query latency, cost per 1,000 queries, and all four core RAGAS scores (faithfulness, answer relevancy, context precision, context recall) get logged per pipeline per query class, with each run repeated enough times to report a median and a spread rather than a single number that variance in LLM output length can quietly distort. The repo ships with the corpus loader, the three pipeline configs, and the evaluation scripts, so the setup is something a team can point at its own documents and rerun, not a chart to take on faith.

That kind of benchmark measures steady-state accuracy and cost. It won’t catch two failure modes that only show up after a GraphRAG pipeline has been running in production for a while, and both are worth planning for before you commit to one:

Extraction errors compound, retrieval errors don’t. A bad vector-RAG retrieval is a per-query event: the next query gets a fresh nearest-neighbor lookup, unaffected by the last one. A bad entity extraction gets baked into the graph. If the index-time model mislabels a relationship or hallucinates an entity, that error persists in every community summary and every traversal that touches it until the corpus is re-indexed. Vector RAG fails query by query. GraphRAG fails structurally, and the failure is harder to spot because it’s wrapped in a fluent community summary rather than a visibly wrong retrieved chunk.

Staleness costs more than it looks like on paper. Full GraphRAG has no cheap path to updating one document. Adding new source material means re-running extraction and re-clustering communities, which is the $33,000-class cost the industry spent two years trying to eliminate, not a one-time expense you pay once and forget. This is the actual reason LightRAG’s incremental merge and LazyGraphRAG’s on-the-fly construction matter beyond their headline indexing cost: they’re what make a graph pipeline viable for a corpus that changes daily instead of one indexed once and queried forever. Before adopting either the original GraphRAG or a lazy variant, ask how often your source corpus changes and confirm the pipeline’s update story matches that cadence, because that number won’t show up in any accuracy benchmark.

None of this argues for or against graphs. GraphRAG-Bench ties on fact lookup and wins by 10 to 13 points on multi-hop and synthesis questions, which means the honest first step is classifying your own query mix before picking an architecture, not adopting a graph because a vendor blog showed a 50-point win rate on a benchmark you don’t share. Run the hybrid-retrieval baseline first. If multi-hop and cross-document questions are a small fraction of real traffic, stop there. If they’re not, the indexing cost that used to make the graph a hard no is gone, and the decision that’s left is entirely about inference: which model runs extraction, how it’s served, and whether it sits next to your graph store and vector store or three network hops away from them.

Here are some resources that you can refer to:

• GraphRAG-Bench: When to use Graphs in RAG (ICLR 2026)
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization,
• LightRAG: Simple and Fast Retrieval-Augmented Generation (EMNLP 2025)
• LazyGraphRAG: Setting a new standard for quality and cost