A compliance officer types one question into the company's RAG chatbot: "Which of our internal controls are affected by the new amendment to Section 12, and what upstream obligations feed into it?" The system returns a confident, well-formatted answer that's missing half the picture, because it found the chunk that mentions "Section 12" and stopped there. It never surfaces the three related obligations that don't mention Section 12 by name but are legally connected to it. The chatbot didn't malfunction. It did exactly what vector search is built to do, find text that reads similarly to the query, which turns out to be a structurally different problem from finding text that's connected to the query.
Why does a well-built vector RAG system still fail on questions like this?
Standard RAG works by chunking documents, embedding each chunk into a vector, and at query time embedding the question and retrieving the chunks whose vectors sit closest to it by cosine similarity. That process is fundamentally a single-hop lookup: it finds text that is semantically close to what you asked, not text that is logically or relationally connected to it. When the true answer requires following a chain, this obligation triggers that control, which in turn was amended by that rule, similarity search has nothing to grab onto once the wording stops overlapping, even though every step in the chain is real and traceable in the source documents.
This is the same chunking-and-embedding mechanism covered in why RAG chatbots give wrong answers: the retrieval step only ever returns what looks like the answer, not what's structurally related to it. Multi-hop questions expose that gap precisely because the connecting fact is never phrased in a way that would embed close to the query.
What does GraphRAG actually do differently?
GraphRAG, developed by Microsoft Research, uses an LLM to extract entities, relationships and key factual claims from unstructured text at ingestion time, then builds those into a knowledge graph, and separately identifies communities within that graph to generate hierarchical summaries of the data's global structure. Instead of retrieving the chunks most similar to a query, GraphRAG retrieval starts from the entities a query touches and traverses the graph's edges outward, pulling in facts connected by relationship, not by wording.
That traversal is the entire difference. A vector index has no concept of "this fact leads to that fact", it only knows distance in embedding space. A graph has an explicit edge for "amends", "triggers", or "depends on", so a query about Section 12 can follow that edge to the obligations it touches even when none of them share a single overlapping phrase with the original question.
Does GraphRAG actually outperform vector RAG, or is that hype?
It depends entirely on the question type, and the data backs that up precisely. GraphRAG-Bench, a benchmark presented at ICLR 2026 spanning 16 disciplines and 20 core textbooks, tested both approaches head to head. On simple fact retrieval, plain text-chunk retrieval scored 60.9% against graph retrieval's 60.1%, effectively a statistical tie. On complex, multi-hop reasoning questions, the gap flips hard: chunks scored 42.9% against the graph's 53.4%, a ten-point win for the graph approach, and on corpus-wide summarization questions the graph reached 64.4% where flat chunk retrieval has no comparable mechanism at all.
The practical nuance matters just as much as the win. As VentureBeat's analysis of the same research puts it, GraphRAG wins the questions that require connecting ideas across a corpus, while plain vector RAG still wins on detail-oriented, single-hop lookups, which means reaching for a graph by default, for every RAG system regardless of question shape, adds real infrastructure and maintenance cost for a benefit that only shows up on a fraction of real queries.
So which one should you actually build?
- Use vector RAG when the answer lives in one or two chunks: policy lookups, product-spec questions, single-document Q&A, anywhere the fact and the question use overlapping language.
- Use GraphRAG when the answer requires connecting entities across separate documents, or when the question is about relationships and global themes rather than a single fact: regulatory mapping, org-wide risk analysis, cross-document dependency questions.
- Budget for the added ingestion cost: graph construction runs an LLM extraction pass over every document to pull entities and relationships before anything can be queried, which is slower and more expensive to build and maintain than a flat embedding index.
- Default to hybrid when real usage mixes both patterns, most production systems do, retrieving from both the vector index and the graph and letting a reranking or synthesis step combine the two rather than betting the whole pipeline on one retrieval strategy.
How AIBOOTSTRAPPER solved this for a client
This exact problem sits at the center of ComplyNexus, the RAG-powered compliance platform we built for a Hong Kong client. Regulatory mapping is inherently graph-shaped: a new rule has to be traced to every control it touches, and every control has to be traced back to the obligations that justify it, a chain that plain similarity search structurally can't follow. The platform continuously ingests regulatory updates, interprets them with an LLM, and maps them against the client's control library with full audit-trail traceability, cutting regulatory change turnaround from three weeks to two hours.
If your RAG system is giving confident but incomplete answers on questions that require connecting facts across documents, that's usually a retrieval-architecture problem, not a prompting one. Our AI product development team can assess whether your use case needs graph retrieval, a better hybrid setup, or just a chunking fix. Book a call to walk through your specific document set.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.