Where vector RAG breaks - and what the fix actually costs
Vector RAG is not broken. Let's be precise about that upfront. For the majority of enterprise retrieval use cases - document Q&A, semantic search over support tickets, summarization of individual contracts - it works well, it's cheap, and it's operationally simple. Don't replace it if it's working.
Quick grounding before we go further, because these terms get used loosely.
Naive / Vector RAG works like a smart search engine. You ask a question, the system converts it and your documents into mathematical vectors, finds the chunks with the closest meaning, and passes those chunks to an LLM to synthesize an answer. Example: a user asks "What are the payment terms in the Acme contract?" - the system retrieves the relevant clause, the LLM reads it and answers. Fast, cheap, and effective for single-document, single-fact lookups. This is what most teams have in production today.
GraphRAG takes a structurally different approach. Before any query runs, it processes your documents into a knowledge graph - a network of entities (companies, people, properties, dates, concepts) and the explicit relationships between them. When a query arrives, instead of retrieving chunks, it traverses that graph. Example: "Which of our enterprise clients have contracts expiring this quarter and are headquartered in EU jurisdictions?" A graph follows the relationships: Client → Contract → ExpiryDate, and Client → Headquarters → Country → EUMember. A vector store sees three separate document fragments with no awareness that they're connected. A graph sees the structure.
But there is a specific, well-defined class of query where it fails systematically. And if your product hits that class, no amount of prompt engineering or chunking optimization will save you.
Here's a representative failure mode. Consider a real estate analytics platform ingesting 40,000 property documents - zoning records, tax histories, HOA agreements, deed restrictions - with a solid vector RAG system: OpenAI embeddings, pgvector, carefully tuned chunking. A user asks:
"Find me properties in the Riverside District zoned R-2 that have had no tax delinquencies in the last 5 years and whose HOA permits short-term rentals."
The system returned confidently retrieved documents that each individually matched a fragment of the query. None satisfied the compound, multi-hop logic of the full question. The model did exactly what the architecture asked it to do. The architecture was wrong for this query type.
Standard semantic search, as formalized by Lewis et al. in the foundational RAG paper, measures a single document's similarity to a single query:
That equation contains the limitation. There is no mechanism for traversing relationships across entities. No mechanism for global aggregation. No mechanism for multi-hop inference - zoning classification → permitted use → rental legality. You retrieve isolated chunks. Not connected knowledge.
Vector RAG is fast, scalable, and cheap to operate. It fails at relational reasoning over structured domain knowledge. Graph-enhanced retrieval solves the relational reasoning problem. It introduces substantial complexity and cost in return. Whether that tradeoff makes sense depends entirely on your query distribution - a point we'll return to.
Why standard GraphRAG doesn't fit SaaS operations
Microsoft Research's GraphRAG paper (Edge et al., arXiv:2404.16130v2) is important work. It demonstrated that for query-focused summarization - broad, synthesizing questions that span many documents - graph-structured knowledge retrieval significantly outperforms naive vector similarity. The architecture generates a knowledge graph from source documents, runs community detection to cluster related entities, and pre-generates summaries per community. Those "community reports" become retrieval targets.
Community detection uses the Leiden algorithm (Traag et al.), which improves on Louvain by guaranteeing internally well-connected communities. The modularity function it optimizes:
Where is the adjacency matrix, and are node degrees, is total edge count, and when nodes share a community. Elegant mathematics.
In production for a dynamic SaaS knowledge base, it's operationally brittle.
The indexing pipeline requires LLM calls for every entity extraction and relationship identification step across the full corpus. On 40,000 documents - not large by enterprise standards - the full GraphRAG indexing pass burns through millions of tokens. At current GPT-4o pricing, you're looking at hundreds of dollars per run on a medium-sized knowledge base.
The structural problem runs deeper than cost. The Leiden algorithm requires the full graph to recompute community assignments. Add a new contract that introduces a new entity relationship and you don't append a node - you potentially invalidate community assignments across a substantial graph partition. For SaaS companies where knowledge bases update continuously - new deals, support tickets, product docs - you need a full nightly re-index that's both expensive and slow.
A note on the cost curve: The divergence shown is real but the absolute numbers are assumption-sensitive. The GraphRAG estimate assumes a per-chunk extraction prompt of roughly 800 tokens, approximately 12 entities per document, and no aggressive batching. Teams using GPT-4o-mini for extraction or batching at 50+ documents per call report costs 3–5x lower than the naive estimate. The curve shape - steep vs. flat - holds regardless of those optimizations. The absolute position of both lines moves. Run your own cost model against your actual document size distribution before treating these numbers as targets.
On the query side, current GraphRAG has also evolved beyond local/global search. The DRIFT search mode (Dynamic Retrieval with Iterative Feature Traversal) is a meaningful addition to the current query story that earlier comparisons miss. DRIFT starts with a global community context - like global search - but then iteratively explores local entity-level evidence to refine the answer, rather than committing to one retrieval level. In practice this makes it substantially cheaper than pure global search (which generates community reports for the entire graph) while being more contextually aware than pure local search. If you are evaluating GraphRAG's query capabilities against LightRAG's hybrid mode, DRIFT is the relevant comparison point, not the original local/global binary.
When Microsoft GraphRAG is the right call: Analytical workloads over corpora with moderate-to-low update frequency. Research summarization, competitive intelligence, legal document analysis. Teams that can invest in configuring incremental indexing correctly and whose query patterns benefit from pre-generated community summaries.
When it's the wrong call: Any SaaS product with continuously ingested content, multi-tenant data, or latency-sensitive retrieval requirements. The operational profile doesn't fit.
When it's the harder call: High-frequency update pipelines (multiple times daily), strict latency budgets, or small engineering teams without capacity to tune incremental indexing configuration. The operational profile is manageable but not simple.
LightRAG: What changes and what it costs you
HKUDS's LightRAG (Guo et al., arXiv:2410.05779v1 · GitHub) keeps the knowledge graph approach but redesigns indexing and retrieval for dynamic datasets. Two mechanisms drive the practical difference.
Dual-Level Vector Keys
Standard GraphRAG retrieves at community summary level - pre-generated text blobs. LightRAG operates at two levels simultaneously:
- Low-level retrieval (entities): Fine-grained vector keys for specific entities and facts. Maps directly to extracted graph nodes.
- High-level retrieval (concepts): Abstract vector keys for synthesized patterns across the graph. Maps to multi-node aggregations.
Every query runs both retrievals in parallel. Results merge before reaching the LLM. The paper calls this "hybrid" mode. It's not a toggle - it's simultaneous. You get entity-level specificity and global context in one pass.
Incremental Subgraph Union
When new content arrives, LightRAG computes a local subgraph for that batch and merges it into the existing graph:
Where , are existing vertex and edge sets, and , are extracted from the new document batch. Entity resolution matches new entities to existing nodes by name and type. New edges write directly. Existing community structures survive unless the new data explicitly contradicts them — only the affected local subgraph updates.
The benchmark numbers from the LightRAG paper are worth reading carefully - but also worth understanding methodologically. Across four evaluation datasets (Mix, Agriculture, CS, Legal) and four quality dimensions (Comprehensiveness, Diversity, Empowerment, Overall), LightRAG in hybrid mode outperforms NaiveRAG, VectorRAG, GraphRAG, and HybridRAG on the majority of metrics. On the Legal dataset - the most structurally complex - hybrid LightRAG achieves the highest scores on all four dimensions. Comprehensiveness over NaiveRAG improves by roughly 20–30 percentage points in win-rate comparisons depending on dataset.
How those win rates are measured: The paper uses pairwise comparison judged by GPT-4 - for each query, the model is shown two anonymized responses from different systems and asked to select the winner per quality dimension. This is a standard and reasonable approach in RAG evaluation, but carries a known limitation documented in the LLM-as-judge literature: GPT-4 tends to favor longer, more comprehensive responses, which can inflate Comprehensiveness scores specifically. Read the benchmark tables with that in mind. The Empowerment and Diversity dimensions are more resistant to this bias and are arguably the more meaningful signals for production use.
The operational number: incremental updates process only new content rather than the full corpus, yielding token cost reductions the paper characterizes as approaching 6000x on dynamic datasets compared to full-reindex GraphRAG. Treat that figure as a directional ceiling, not a guaranteed production outcome. The paper benchmarks under controlled conditions - fixed corpus, clean entity extraction, consistent document length. Real-world variance is significant: extraction failure rates, entity resolution mismatches, and corpus update patterns all affect the actual ratio. Informal practitioner reports in the LightRAG and GraphRAG GitHub issue trackers, and engineering post-mortems shared in public forums, typically describe real-world cost reductions in the 200x-1500x range for dynamic datasets - still a meaningful advantage, but not the headline number. No controlled third-party benchmark has published a definitive figure at the time of writing; treat any specific ratio as an order-of-magnitude estimate until you measure against your own corpus.
For 1,000 new documents per day, that's the difference between a $200/day indexing bill and somewhere between $0.15 and $1.00 - depending on your specific corpus and update pattern.
What LightRAG doesn't give you: The pre-generated community summaries that make GraphRAG genuinely better at broad, global summarization queries. If your primary use case is "summarize everything we know about Topic X across 50,000 documents," GraphRAG's community reports are still the stronger retrieval signal. LightRAG trades some of that global synthesis quality for dramatically lower operational cost and update latency. Know which tradeoff you're making.
And what you should consider before defaulting to LightRAG at all: The diagram above makes graph-based retrieval look like the natural next step after vector RAG. It isn't necessarily. Two alternatives worth benchmarking first:
- Query decomposition pipelines: Break a complex multi-hop query into sequential sub-queries, answer each independently against your vector store, then merge. No graph required. For a significant portion of queries that appear relational, this approach resolves them adequately at a fraction of the infrastructure cost. Tools like LangChain's query decomposition chains and DSPy's multi-hop modules implement this.
- Agentic retrieval: Let the LLM iteratively decide what to retrieve next based on intermediate results. Recent benchmarks on multi-hop QA datasets show agentic systems competitive with or outperforming pre-built graph approaches on several reasoning tasks, with significantly lower upfront indexing investment.
LightRAG is a well-validated, production-viable option. It is not the only option for multi-hop reasoning. If your team is evaluating it seriously, run it against a well-tuned query decomposition baseline on your actual queries before committing to the graph infrastructure.
AWS Enterprise deployment: Neptune vs. Neo4j
Once you commit to graph-enhanced retrieval at scale, you need a graph database that handles enterprise read throughput without becoming a bottleneck.
This section covers the two most common production choices. They are not the only options: Weaviate and Qdrant both offer hybrid vector-plus-graph or vector-plus-filter architectures that suit simpler relational use cases without the operational overhead of a dedicated graph database. If your graph is shallow (1–2 hops, limited entity types), evaluate those before committing to Neptune or Neo4j.
AWS Neptune
Neptune separates compute from storage. The storage layer replicates six ways across three availability zones. Compute nodes share the storage pool - read replicas scale horizontally without data copying. During LightRAG's incremental update cycle, the writer processes subgraph union operations while read replicas continue serving queries against existing graph state. No write-lock contention affects read latency. At millions of traversals per day, this matters.
Neptune supports both Property Graph (openCypher, Gremlin) and RDF/SPARQL natively. For domains with strong ontological requirements - healthcare, legal, financial services - RDF isn't a curiosity, it's the right modeling primitive.
The hard constraint: Neptune is AWS-locked. Multi-cloud organizations should treat this as a first-order concern, not a footnote.
Neo4j with causal clustering
Neo4j's causal clustering guarantees that a write acknowledged to the indexing client is visible to all subsequent reads from that client, regardless of which replica serves them. For RAG systems where the indexing pipeline and query path run on separate infrastructure, this eliminates a class of subtle consistency bugs where a freshly indexed entity isn't visible to an immediate subsequent query.
Neo4j ships the Graph Data Science library natively - Leiden algorithm, PageRank, community detection. If you run periodic full re-clustering alongside daily incremental updates (a reasonable hybrid strategy for large corpora), GDS handles it without additional infrastructure.
The operational cost: Causal clustering requires managing leader election, Raft consensus, and replica lag. It rewards teams with distributed systems depth. It punishes teams without it.
Orchestrating the state machine with LangGraph
The LightRAG pipeline is stateful and cyclical. It doesn't fit a linear chain.
You have document ingestion, entity extraction (LLM calls per batch), entity resolution against the existing graph, subgraph union writes, vector index updates, then the query path: dual-level parallel retrieval, context fusion, generation. The resolution step loops on failures. Graph writes and vector writes can diverge and need rollback. Long-running indexing jobs fail mid-pipeline and need to resume, not restart.
LangGraph handles this cleanly. Each step becomes a node in a directed graph. Conditional edges handle branching. Persistent checkpointing (DynamoDB or PostgreSQL backend) means pipeline failures resume from the last committed state.
LangGraph is not the only option here. Temporal and Prefect provide similar stateful workflow orchestration with durable execution and are a reasonable choice if your team already operates them. The key requirement is durable state between steps and conditional branching — any orchestration system that provides those two properties works. LangGraph has the advantage of native LLM-aware abstractions and a tighter integration with LangChain's ecosystem, which reduces glue code if you're already in that stack.
The
GraphRAGState object carries the document batch, extracted entity list, resolution mappings, and accumulated subgraph delta - all checkpointed between steps. Network partition drops the graph write at step 4, the next run picks up after step 3. No re-extraction. No duplicate LLM calls.Latency and complexity tradeoffs: the numbers you actually need
Cost gets discussed. Latency rarely does. Here are representative figures across retrieval approaches, based on reported benchmarks and production community observations. Treat these as planning ranges, not guarantees - your corpus size, hardware tier, and query complexity will shift them.
Approach | P95 Query Latency | Graph Traversal Overhead | Daily Update Cost (40K docs) | Infra Complexity |
Vector RAG (pgvector / Pinecone) | 50-150ms | None | ~$0 | Low |
LightRAG - entity mode | 100-250ms | +40-80ms | ~$0.03-$1.00 | Medium |
LightRAG - hybrid mode | 200-500ms | +80-150ms | ~$0.03-$1.00 | Medium |
Microsoft GraphRAG (full re-index) | 300-800ms | +150-400ms | $150-$400 | High |
Hybrid router (vector + graph) | 50-500ms (path-dependent) | Per-path overhead | ~$0.05-$2.00 | High |
Sourcing note on this table: P95 latency figures for vector RAG are derived from published pgvector and Pinecone benchmarks at ~10M vectors on comparable cloud hardware. Graph traversal overhead is estimated from Neptune and Neo4j documentation on 2–3 hop traversals at equivalent scale. LightRAG-specific latency figures are extrapolated from the paper's reported query times on test hardware and should be validated against your own deployment. No single controlled benchmark covers all five rows under identical conditions - treat this as a planning framework, not a specification.
The latency overhead of graph traversal is real but not prohibitive. A hybrid-mode LightRAG query at P95 of 500ms is acceptable for most enterprise search interfaces. It's not acceptable for autocomplete or real-time suggestions - those paths should always stay on vector retrieval.
The infra complexity column is the one that kills projects. "High" here means: graph database cluster operations, vector index sync, entity resolution pipeline, stateful orchestration, and ReBAC enforcement - all running correctly and consistently under load. Don't underestimate the operational burden on your on-call rotation before committing to the full stack.
Securing it: ReBAC for Multi-Tenant GraphRAG
Here's the security failure mode that almost every team misses until it becomes an incident: knowledge graphs aggregate information in ways that leak data across tenant boundaries.
In vector RAG, you scope retrieval with a
tenant_id metadata filter. Simple. In a knowledge graph, a node representing a shared vendor entity might have edges to documents owned by both Tenant A and Tenant B. A traversal from Tenant A's query context can follow those edges into Tenant B's data. The model will synthesize an accurate answer from information the user was never authorized to see. You're not hallucinating. You're accurately answering a question you shouldn't have been able to answer.You need Relationship-Based Access Control. Google's Zanzibar system formalized this at scale; the same principles apply here.
The model is a tuple store:
(object, relation, subject).(document:contract_447, owner, tenant:acme_corp)
(document:contract_447, viewer, user:alice@acme.com)
(graph_node:vendor_xyz, readable_by, tenant:acme_corp)
The check function
check(user, relation, object) traverses the relationship graph to determine if access exists, including through inheritance.For GraphRAG, enforce this at two layers:
Layer 1 - Pre-retrieval graph scoping. Before traversal, the query is decorated with the tenant's permission set. The graph database query physically scopes traversal to nodes where
(node, readable_by, tenant_id) exists. The traversal cannot reach unauthorized nodes — they are excluded at the query layer.Layer 2 - Post-retrieval context filtering. Before the retrieved subgraph reaches the LLM, every node is validated against the current user's permissions. Nodes failing the check are redacted. The model never receives unauthorized data.
Layer 1 is the primary access boundary and a performance optimization. Layer 2 is the safety net for edge cases in complex traversal queries. Both layers are required.
The MDPI clinical RAG evaluation is instructive here: their analysis of RAG variants in healthcare settings found that retrieval accuracy alone is insufficient for regulated domains - the retrieval boundary is a compliance and safety issue. One cross-tenant retrieval failure in a B2B SaaS product is a breach, not a bug.
The actual decision you have to make
This is not a vector vs. graph decision. That framing is outdated and unhelpful.
The real decision is: what does your query distribution look like, and what is the right retrieval component for each query class?
Modern production RAG systems in 2026 are hybrid orchestration systems. Vector retrieval handles the high-volume, single-entity, fast-path queries. Graph retrieval handles the relational, multi-hop, aggregation queries. The router - whether rule-based, classifier-based, or LLM-based - directs each incoming query to the right path. Neither component is universally superior. Both are required for a complete system.
A note on the query distribution diagram above: The traffic split shown (roughly 60% vector / 30% graph / 10% global) is design intuition based on typical enterprise query patterns, not empirical data from a specific deployment. Your actual distribution depends on your domain, user base, and product surface. Before making routing architecture decisions, instrument your existing system: log and classify a representative sample of 1,000 real queries by type. Let that data drive your routing thresholds, not industry estimates including this one.
Before any of this: benchmark against a strong vector baseline. Advanced vector pipelines with query decomposition, cross-encoder reranking, and HyDE (Hypothetical Document Embeddings) close the gap on many query types that intuitively seem to require a graph. If a well-tuned vector pipeline reaches 85% of the quality you need at 20% of the operational complexity, that may be the right call for your team's current scale and staffing. Don't add graph infrastructure to solve a problem your existing system hasn't actually failed at.
Start with LightRAG's incremental indexing model if your knowledge base updates daily and you're operating at SaaS scale - and if you've already confirmed that vector retrieval meaningfully fails on your query distribution. The token cost profile is viable. The codebase (HKUDS/LightRAG) is clean. Dual-level retrieval covers most relational query types without requiring full community pre-generation.
Layer in Microsoft GraphRAG's community summaries for the global summarization path only if your benchmarking shows those queries are a meaningful portion of your traffic and vector retrieval measurably fails on them. Don't add the operational cost speculatively.
Choose your graph database based on operational team depth, not benchmarks. Neptune if you're AWS-native and want managed infrastructure. Neo4j if you need graph data science primitives and have distributed systems experience to back it up.
Implement ReBAC from day one. Retrofitting access control onto an existing knowledge graph is significantly harder than building it in. Every node written to your graph needs a corresponding owner tuple in the permission store from the moment it's created.
Orchestrate with LangGraph. The pipeline is too stateful for a linear chain. You will hit resolution failures, vector/graph state divergence, and partial rollback scenarios that a proper state machine handles and a chain does not.
The teams doing this well aren't choosing between retrieval paradigms. They're building routers that use each paradigm where it's strongest, with the operational controls to keep the whole system secure and cost-effective. That's the actual engineering problem. It's harder than picking a winner. It's also the right problem to solve.
