Your RAG Chatbot Isn't Broken. Your Chunking Strategy Is.

Why a bad chunk boundary breaks retrieval even with a great embedding model

A RAG pipeline works by breaking source documents into chunks, converting each chunk into a vector with an embedding model, and at query time, embedding the user's question and finding the chunks whose vectors sit closest to it by cosine similarity. That pipeline only works if a chunk's vector actually represents one coherent idea. When a chunk boundary lands mid-sentence, or worse, mid-clause, the embedding model is forced to compress two unrelated fragments into a single vector, and that vector ends up sitting in an ambiguous region of the embedding space, close to neither concept.

This is a structural failure, not a tuning problem. A better embedding model still has to embed whatever text it's handed, and if the text handed to it is an incoherent fragment, the resulting vector is incoherent too. The retrieval step then has no coherent match to find, because the passage a user actually needs was never stored as a single retrievable unit in the first place.

The chunking tradeoff the benchmarks actually show

There are two dominant chunking strategies, and they fail in opposite directions. Recursive character splitting cuts text at natural boundaries (paragraphs, then sentences, then words) at a fixed target size, while semantic chunking uses an embedding model to detect topic shifts and splits there instead. A benchmark comparison in Denser AI's 2026 chunking strategy guide found recursive splitting reaching 69% end-to-end accuracy versus semantic chunking's 54%, yet semantic chunking actually retrieved more precisely, 91.9% recall versus recursive's 88%.

That split result matters more than either number alone: semantic chunking is genuinely better at finding the right passage, but it tends to produce fragments too short to carry enough context for the model to generate a correct answer from, once retrieved. Recursive splitting produces chunks that are less precisely bounded but more self-contained, which is why it wins on the metric that actually reflects what the user sees.

The size floor and ceiling most teams get wrong

• 50 to 100 tokens: too small. Chunks this short are frequently missing the context needed to answer a question on their own, even when retrieval finds the right one.
• 256 tokens: a reasonable target for short Q&A style content, where each chunk maps closely to one discrete fact.
• 512 tokens: the default sweet spot for most production RAG systems handling general documents, per Denser AI's benchmark guide.
• 1,024 tokens: appropriate for long-form legal or technical documents where a single clause depends on surrounding context to be interpreted correctly.
• 1,000+ tokens as a default for everything: too large. Cramming multiple ideas into one chunk dilutes the embedding signal, so the vector represents an average of several concepts instead of a precise match for any one of them.

The fix that actually moves the number: give each chunk its context back

The single highest-leverage fix isn't picking the perfect chunk size, it's restoring the context a chunk loses the moment it's separated from its source document. Anthropic's contextual retrieval technique prepends a short, chunk-specific summary, generated by a cheap model, describing what the chunk is about and where it sits in the source document, before it's embedded. A chunk that used to just say 'the fee increases by 5% annually' becomes something like 'this chunk is from the Q2 2026 pricing addendum for the Acme contract; the fee increases by 5% annually starting in year two.'

That one change cut retrieval failure rates by 49% on its own, and by 67% when combined with a reranking step that re-scores the top candidates before they reach the model, according to Anthropic's published benchmarks. It works because the embedding no longer has to represent an orphaned fragment, it represents a fragment that knows what document and section it came from, which is exactly the information a naive fixed-size chunker throws away.

What actually changes chunk overlap: newer data than most guides cite

Chunk overlap, repeating the last 50-100 tokens of one chunk at the start of the next, is standard advice for catching a concept that spans a chunk boundary, and Denser AI's guide still recommends a 50-token overlap as a safe hybrid-system default. But a more recent systematic analysis using SPLADE retrieval, run in January 2026, found overlap provided no measurable benefit for that setup and only added indexing cost. The takeaway isn't that overlap is universally useless, it's that the right chunking configuration depends on the retrieval method underneath it, and a default copied from a two-year-old blog post may no longer match how your specific pipeline retrieves.

How AIBOOTSTRAPPER solved this for ComplyNexus and ComplySpark

AIBOOTSTRAPPER built exactly this retrieval discipline into ComplyNexus, a RAG-powered compliance engine that continuously monitors regulatory sources, interprets new rules with an LLM, and maps them to a client's control library. A compliance answer built from a broken chunk boundary isn't a minor inconvenience, it's a wrong regulatory interpretation an auditor could catch, so the chunking, metadata and context discipline covered in this guide was part of the build from day one, not a patch added after the retrieval quality complaints started. The result: a three-week regulatory change turnaround cut to two hours, with 92% less manual review time.

The same foundation carries into ComplySpark, a document-drafting copilot grounded in a client's own policy library, which depends on the same chunk-level context to guarantee zero off-policy language slip-ups while cutting document drafting time from days to an afternoon. Full results for both are on the case studies page.

How AIBOOTSTRAPPER helps

AIBOOTSTRAPPER's AI product development team audits and rebuilds RAG pipelines that are technically working, embeddings generated, vectors indexed, queries returning results, but quietly wrong on the specific answers a business actually depends on. That's almost always a chunking and context problem sitting underneath a model choice everyone assumed was the issue.

If your chatbot is confidently citing the wrong clause or the wrong paragraph, book a call and we'll look at how your documents are actually being chunked before recommending a different model.

Sources and further reading

1. 1.Anthropic — Introducing Contextual Retrieval
2. 2.Denser AI — RAG Chunking Strategies 2026: 8 Methods Compared