RAG Evaluation: Find the Evidence, Use It, and Cite It

September 2026 · 14 min read

Before you start: pipeline-stage diagnosis and RAG fundamentals.

A RAG answer can retrieve the right document and lose the crucial sentence during truncation. It can cite a real page that does not support its claim. It can faithfully repeat an outdated rule. A single answer score conceals these different failures.

Follow one question through the pipeline: “What are the return windows for standard and sale items, and can I return a sale item after 20 days?” Our fictional store allows standard returns within 30 days and sale returns within 14 days. Assume all other return conditions are satisfied. The correct decision is no: 20 days exceeds the sale window.

Ask whether the evidence is sufficient for this question

Annotate two evidence requirements: the standard-item window and the sale-item window. The question supplies the purchase age, so it needs no retrieved passage. The two windows are required because the question explicitly asks for both. For a question asking only whether the sale item qualifies, the sale rule and purchase age would suffice.

A requirement is a fact needed to answer, not a chunk ID. Several chunks may express the same fact; retrieving duplicates should not multiply coverage. Conversely, one passage may cover several requirements. Record which passages satisfy which requirements, including document revision and the relevant span.

Requirement recall is the number of covered requirements divided by the number annotated for the question. With G and E retrieved, it is 2/2 = 100%. If the context builder drops E, final-context recall is 1/2 = 50%. A retriever-level success has become a context-level failure.

Our chunk precision is useful supplied chunks divided by all supplied chunks. G alone has 100% precision but only 50% requirement recall; G, E and D have 100% recall and 2/3 precision. Neither fraction alone establishes a good context. These are explicit, hand-annotated diagnostics; chunk splitting and the judgment unit affect the precision denominator.

Track token cost as well. Equal chunk counts can occupy very different context budgets, and a tiny exception can matter more than a long general policy. Save retrieved IDs and scores, filtering decisions, selected spans, their order and the exact final context. Check permissions and revision applicability before calling evidence usable.

Correctness asks about the task; faithfulness asks about the context

Correctness compares the answer with the accepted task facts. Here the current rules and the user’s 20-day purchase age define those facts. Faithfulness asks whether the answer’s claims follow from the context the generator received, together with facts supplied in the question.

An old sale policy allowing 30 days can support “yes, 20 days qualifies,” even though that answer is wrong under the current 14-day policy. In this stale-source example, the generator receives the old rule without a revision warning; the evaluator knows it has been superseded. The reverse is possible too: the model might answer 14 days from memory while receiving no passage that establishes it. Correctness and context support need separate labels.

A working reference can still point to the wrong evidence

Split the answer into three claims: the standard window, the sale window and the decision for this purchase. For each claim, ask whether it is correct, whether the supplied context supports it, and whether its attached citation supports it. Click a claim below to inspect the evidence behind its labels.

In “Wrong citation,” the sale-window claim is correct and E supports it in the supplied context, but its citation points to G. The reference resolves successfully; it does not establish the sale rule. In “Uncited decision,” the conclusion follows from E and the question, but the answer supplies no citation for that conclusion.

Keep three citation checks distinct. Validity checks that the reference resolves to the intended document and revision. Support checks the cited material against the associated claim. Completeness checks that claims requiring evidence actually receive supporting citations. Our “cited support” count is the fraction of the three claims that meet that last requirement; an uncited claim fails it.

For claims that require several passages, evaluate their combined evidence. Extra citations can be irrelevant even when the combined set supports the sentence. ALCE formalizes statement-level citation recall and citation precision, including the role of multiple citations. State your own segmentation and aggregation rules when comparing implementations.

Finally, judge task completeness. An answer that states only the standard rule can have perfect support for its one claim while failing to answer the sale question. Many easy supported claims must not hide an incorrect consequential decision. Record a task-level success judgment alongside claim counts and the severity of errors.

Use controlled runs to decide where to repair the system

For a failed question, keep the model, prompt template and generation settings matched while changing the evidence supplied. Record the context change explicitly, and repeat stochastic generations when variability matters.

If G and E enable a correct answer but the actual context does not, inspect evidence selection and context construction first. The gap does not isolate the embedding model. If the answer still fails with verified sufficient evidence, inspect instruction following, interpretation of exceptions, arithmetic and the task rubric. Adding more retrieved chunks may leave all of those failures intact.

Measure what happens when evidence is insufficient

An all-answerable dataset cannot show whether the system invents an answer when support is missing. Include insufficient-evidence questions and define the useful fallback: a qualified partial answer, a clarification request or an explicit inability to determine the answer. Test stale or conflicting sources and document instructions that attempt to override the task as separate slices.

The next example contains ten fixed candidate answers. Seven questions have sufficient evidence; three do not. Each candidate has a made-up confidence score, which is not a calibrated probability. Answer only when its score meets a threshold:

Answer coverage divides emitted answers by all ten questions. Correctness among answered questions divides correct emitted answers by emitted answers. Also track how many of the three insufficient-evidence questions receive an answer. The denominators address different concerns and stay visible in the figure.

At threshold 0.90, both emitted answers are correct, but coverage is only 20%. Abstaining on everything gives zero coverage and undefined answered-only correctness, not a perfect score. At threshold 0.80, a high-scoring insufficient-evidence answer still gets through. A confidence cutoff is not proof of grounding.

Choose thresholds on validation data with an explicit cost for wrong answers and missed useful answers. Report coverage beside quality and evaluate the insufficient-evidence slice separately. The appropriate tradeoff depends on the application; no single threshold wins from these ten artificial cases.

Evaluate the evaluator with a readable rubric

Automated judges make evaluation cheaper, but their labels are measurements with errors. RAGAS separates faithfulness, answer relevance and context relevance using automated metrics. ARES combines learned judges with human annotations to address evaluation uncertainty. Neither makes a domain-specific rubric unnecessary.

Give a faithfulness judge the question, answer and actual supplied context. Give a correctness judge trusted current reference facts as well. For citation support, resolve the cited spans rather than supplying an unrelated gold passage. Otherwise the judge may credit evidence the system never used or never cited.

{
  "question_id": "returns-sale-20-days",
  "claim": "Sale items have a 14-day return window.",
  "supplied_context_ids": ["G-current", "E-current"],
  "cited_span_id": "G-current:standard-window",
  "correct_against_current_rules": true,
  "supported_by_supplied_context": true,
  "supported_by_attached_citation": false,
  "reason": "E states 14 days; the cited G span concerns standard items."
}

Audit a representative sample with human reviewers, including errors, exceptions and difficult slices. Inspect disagreements by dimension, not just overall agreement. Hide system identity and swap pairwise answer order to check order sensitivity; the MT-Bench judge study documents position and other biases. Invalid judge outputs should be recorded rather than silently converted into passing scores.

Compare frozen systems on the same questions and use the paired uncertainty methods already developed in this series. Report requirement coverage, task success, citation support, abstention behavior and latency with their slice counts. Link failures back to the saved trace so the next change targets the stage that actually failed.

References and further reading

• Es et al.: RAGAS — automated measures of faithfulness, answer relevance and context relevance; our fixed requirement counts are an explicit annotation scheme.
• Gao et al.: ALCE — answer correctness, citation recall and citation precision, with multi-passage support and irrelevant-citation checks.
• Saad-Falcon et al.: ARES — component judges and human annotations for evaluation with prediction errors.
• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — judge agreement, answer-order effects and other evaluator biases.