Why Your RAG Chatbot Gives Wrong Answers: How Embeddings, Cosine Similarity and Reranking Actually Work

What an embedding actually is

An embedding is a list of numbers, typically 768 to 3,072 of them, that represents the meaning of a piece of text as a point in high-dimensional space, produced by a model trained specifically so that texts with similar meaning end up as nearby points. OpenAI's embeddings guide documents this directly: the same embedding model that turns 'refund policy' into a vector will turn 'return policy' into a very close but not identical vector, because the model learned that these phrases mean similar things, without either phrase sharing a single word.

This is the entire trick behind semantic search: instead of matching keywords, you are comparing positions in a meaning-space, so a query using different words than the source document can still retrieve it, which keyword search structurally cannot do.

How cosine similarity picks the 'closest' chunk

Cosine similarity measures the angle between two vectors, not their distance, which matters because it makes the comparison insensitive to how long a piece of text is, only to what direction it points in meaning-space. A score of 1.0 means the vectors point in exactly the same direction (near-identical meaning), 0 means unrelated, and vector databases like Pinecone and Weaviate compute this score between your query's embedding and every stored chunk's embedding to return the highest-scoring matches, exactly as described in vector search engines' core similarity search documentation.

The failure mode is that 'highest cosine similarity' is not the same as 'the correct answer', it is only 'the most semantically similar stored chunk'. If your policy library has both the current and the superseded version of a document, and both chunk similarly to the query, cosine similarity alone has no concept of recency or authority, it will happily return whichever one embeds slightly closer, which is exactly how a compliance bot cites an outdated policy with full confidence.

Where naive chunking silently breaks retrieval

• Chunking mid-sentence or mid-table: splitting a document into fixed 500-token blocks without regard for structure routinely cuts a table's headers away from its rows, or a clause away from the exception that qualifies it, so the retrieved chunk is technically 'similar' but missing the context that changes its meaning.
• No metadata filtering before similarity search: if superseded documents are not tagged and excluded before the cosine similarity comparison runs, an outdated chunk competes on equal footing with the current one, and nothing in the similarity score itself signals which is authoritative.
• Chunks too large, diluting the embedding: a 2,000-token chunk covering five unrelated topics produces a single 'averaged' embedding that represents none of those topics precisely, so it scores mediocre-but-not-zero against many unrelated queries instead of scoring high against the one query it should match.
• Chunks too small, losing context: a 50-token chunk may embed a sentence fragment accurately but strip away the surrounding qualifier ('...unless the claim exceeds $500'), so the model receives a technically-retrieved but practically-misleading piece of text.

How reranking fixes what similarity search alone misses

Reranking adds a second, more expensive but more accurate pass after the initial vector search: instead of comparing embeddings, a reranking model reads the actual query and each of the (typically 20-50) candidate chunks returned by the first pass, together, and scores how well each one actually answers the query, closer to how a human would judge relevance. Services like Cohere's Rerank implement exactly this as a drop-in second stage.

This two-stage design, cheap broad retrieval by cosine similarity followed by expensive precise reranking on a small candidate set, is standard in production RAG systems for a reason: reranking alone over your entire document set would be too slow and expensive, but running it only on the top 20-50 candidates already narrowed by similarity search is fast and meaningfully more accurate, and it is usually the single highest-leverage fix for a RAG system that retrieves 'close but wrong' chunks.

How AIBOOTSTRAPPER built this for clients: ComplyNexus and ComplySpark

AIBOOTSTRAPPER built exactly this retrieval discipline into ComplyNexus, a RAG-powered compliance engine that continuously monitors regulatory sources, maps new rules to a client's control library, and surfaces gaps with full audit trails, cutting a 3-week regulatory change turnaround down to 2 hours with 92% less manual review time.

The same foundation powers ComplySpark, a document-drafting copilot grounded in a client's own policy library, which had to solve precisely the superseded-document problem described above to guarantee zero off-policy language slip-ups while cutting document drafting time tenfold. Both systems depend on the metadata filtering, chunking discipline and (where needed) reranking covered in this guide, not on cosine similarity alone. Full results for both are on the case studies page.

How AIBOOTSTRAPPER helps

If your RAG system is retrieving plausible-sounding but wrong chunks, the fix is rarely 'use a bigger model', it is almost always in the retrieval layer: chunking strategy, metadata filtering for recency and authority, and whether a reranking stage exists at all.

AIBOOTSTRAPPER's AI product development team audits and rebuilds retrieval pipelines exactly like the ones behind ComplyNexus and ComplySpark, for compliance, support and internal knowledge use cases. Book a call if you want a second pair of eyes on why your RAG system keeps citing the wrong source.

Sources and further reading

1. 1.OpenAI — Embeddings guide
2. 2.Pinecone — Vector search fundamentals
3. 3.Cohere — Rerank overview