The demo went perfectly. The team asked the new support chatbot twenty questions from the help centre, it answered all twenty correctly, and it shipped on Friday. By Wednesday a customer had been told the refund window was 60 days when the policy says 30, and the head of support wanted to know why. The engineers tried the obvious moves: reword the system prompt, bump the model to a bigger one, raise the number of retrieved chunks from four to eight. The bug moved around but never died, and nobody could say whether any change had helped, because the only test the team owned was a human reading answers and deciding whether they looked right. A RAG system is two systems chained together, a retriever and a generator, and a single eyeballed score cannot tell you which one just failed.
Why is 'does the answer look right' the wrong test for a RAG system?
Because a wrong answer has at least two completely different root causes that need opposite fixes, and a single end-to-end judgement can't separate them. Either the retriever never surfaced the chunk containing the right fact, in which case the model was answering blind, or the retriever did surface it and the generator ignored, misread or embellished it. Rewording the prompt cannot fix the first failure, and swapping the embedding model cannot fix the second.
That is why a production RAG evaluation measures the two stages separately: retrieval metrics score what came back from the vector store, generation metrics score what the model did with it. Once you have both, every bad answer lands in a diagnosis instead of a guessing game. The rest of this post builds that harness, starting with the retrieval side, which is where most RAG chatbots that give wrong answers actually break.
What do recall@k and context precision actually measure?
Recall@k asks one question per test query: did at least one relevant chunk appear in the top k results the retriever returned? Set k to the number of chunks you actually place in the prompt, not an arbitrary 10, because a relevant chunk ranked eleventh is invisible to a model that only ever sees the top five. If 88 of your 100 test queries have a relevant chunk in the top five, your recall@5 is 0.88, and the other 12 queries are failures no prompt engineering can rescue.
Context recall and context precision, as defined in the Ragas metric documentation, refine that. Context recall is the number of claims in the reference answer that are supported by the retrieved context, divided by the total claims in the reference answer, so it tells you whether everything needed to write the right answer was retrieved, not just one lucky chunk. Context precision rewards ranking relevant chunks above irrelevant ones: in the documentation's own example, putting an irrelevant chunk first drops the score from roughly 1.0 to 0.5, while placing it second barely moves it. Precision is the metric that tells you whether a reranker is earning its latency.
How do you tell a retrieval failure from a generation failure?
Read the metric pattern, not the individual score. Each combination points at a different layer of the pipeline and a different fix, which is the entire payoff of splitting the evaluation in two.
| Metric pattern | What it means | Where to fix it |
|---|---|---|
| Low recall@k or low context recall | The right evidence never reached the model | Chunking strategy, embedding model, hybrid BM25 plus vector search, metadata filters |
| High recall, low context precision | The right chunk is retrieved but buried under noise | Add a reranker, lower k, tighten filters |
| High recall and precision, low faithfulness | The model had the evidence and didn't stay inside it | Grounding instructions, mandatory citations, lower temperature, stronger generator |
| Everything high, users still get wrong answers | The source document is wrong, stale or the reference answer is wrong | Content freshness pipeline, golden-set audit |
Diagnostic matrix for RAG evaluation: retrieval-stage metrics on the first two rows, generation-stage on the third.
What is faithfulness, and why can a perfect score still be a wrong answer?
Faithfulness measures whether the answer's claims are supported by the retrieved context. The Ragas faithfulness definition works in three steps: break the response into individual statements, check whether each statement can be inferred from the retrieved context, then divide supported claims by total claims. In its worked example, a response stating that Einstein was born in Germany on 20 March 1879 when the context says 14 March 1879 scores 0.5, because one of the two claims is supported and the other isn't.
Read that definition closely, though: faithfulness scores agreement with the retrieved context, not agreement with reality. If the retriever pulled a superseded 60-day refund policy and the model repeated it accurately, faithfulness is 1.0 and the customer is still misinformed. That is the reason faithfulness has to be paired with retrieval metrics and a source-freshness check, never used alone. A faithful answer built on the wrong chunk is a retrieval bug wearing a generation score.
How do you build a golden dataset that actually catches regressions?
A golden dataset is a fixed set of questions, each paired with a reference answer and the ID of the chunk or chunks that contain the evidence. Recall@k and context recall need those chunk IDs or reference answers as ground truth, which is the part teams skip because it takes human time. Here is the structure we build, offered as our working practice rather than a published standard:
- **Mine real queries, not invented ones.** Sample from actual support tickets or chat logs, stratified by intent, so the set mirrors what customers really ask, including the badly phrased ones.
- **Label the evidence, not just the answer.** Record which source chunk IDs a correct answer needs, so you can score retrieval independently of generation.
- **Include unanswerable questions.** A slice of queries whose correct behaviour is to say the knowledge base doesn't cover this, because refusal is a behaviour you have to test on purpose.
- **Include multi-chunk questions.** Queries needing two or three separate chunks are where context recall diverges from simple recall@k.
- **Freeze and version it.** Change the dataset deliberately and log why, otherwise you can't compare last month's scores with today's, the same discipline used to validate a replacement when an LLM model version gets deprecated.
Why does your LLM judge give unreliable faithfulness scores?
Because faithfulness and answer-quality scoring are themselves usually performed by an LLM, and LLM judges carry measurable biases. The CALM study, 'Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge', catalogues 12 distinct biases, including position bias, verbosity bias, self-enhancement, authority bias and sentiment bias, and the project's results page reports that self-enhancement vulnerability showed up across all six models tested and that robustness on chain-of-thought-related bias sat between 0.651 and 0.745. A judge that quietly prefers longer answers, or answers written by its own model family, will reward exactly the wrong changes to your pipeline.
You can't remove judge bias, but you can contain it:
- **Use a different model family as judge than as generator**, so self-enhancement can't inflate the score for your own outputs.
- **Prefer decomposed binary checks over 1-to-10 ratings.** Faithfulness works because each claim gets a supported-or-not verdict; scalar 'rate this answer' prompts are where verbosity and sentiment bias creep in.
- **Calibrate against humans.** Have a person label a sample of 30 to 50 items, compare judge to human, and track the agreement rate; if it drops after a judge-prompt change, the change is the bug.
- **Pin the judge model version and temperature.** An unpinned judge that changes underneath you creates score drift that looks like a pipeline regression, a variant of the run-to-run variance problem with temperature.
How do you run this continuously instead of once before launch?
Treat the evaluation like a test suite that gates changes. Any edit to chunk size, the embedding model, the reranker, the prompt or the generator model triggers a run over the golden dataset, and the build fails if a metric regresses beyond a tolerance you set against your own baseline. Gate on regression from baseline rather than an absolute number, because there's no universal 'good' recall@k; the right bar depends on how costly a wrong answer is in your domain.
Log the retrieved chunk IDs on every production query too. That gives you a stream of real traffic to sample into the golden set each month, and it turns 'the bot said something wrong' from an anecdote into a retrievable trace. This is the RAG-specific layer under the trace-based checks we describe in how to evaluate an AI agent before it ships.
How AIBOOTSTRAPPER solved this for a client
When we built ComplyNexus, a RAG-powered compliance engine for a Hong Kong regulatory team, a plausible-sounding but unsupported answer was not a cosmetic bug, it was an audit finding. The system continuously monitors regulatory sources, has an LLM interpret new rules, maps them to the client's control library, and surfaces gaps with full audit trails. The delivered outcome was 100% audit-ready traceability, regulatory-change turnaround cut from three weeks to two hours, and 92% less manual review time.
That traceability requirement is the same property a faithfulness check enforces at scale: every claim in an answer has to point back to a retrieved source. We design retrieval and grounding checks into RAG builds from the start rather than retrofitting them after a wrong answer reaches a customer. If your chatbot is live and you can't say whether the last prompt change made it better or worse, talk to us about setting up a RAG evaluation harness, or see our AI product development services.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.