Evaluating RAG: Metrics, Golden Sets, Experiments, and Results

“The answers look better” is not an evaluation. Enterprise RAG needs a versioned test program that identifies which stage improved, which user segment regressed, and whether the gain is worth its latency and cost.

Build the dataset from real work

Start with anonymized search logs, support questions, subject-matter-expert interviews, and known failure cases. Add synthetic questions only to fill explicit coverage gaps; synthetic-only sets are usually cleaner than production traffic and inherit the generating model’s biases.

Each case should contain:

{
  "id": "travel-042",
  "question": "What is the meal limit for a Paris employee visiting Tokyo?",
  "language": "en",
  "principal_fixture": "employee-fr",
  "reference_answer": "...",
  "relevant_documents": ["travel-policy-v8"],
  "relevant_spans": [{ "document": "travel-policy-v8", "section": "4.2" }],
  "must_include": ["currency", "daily limit"],
  "must_not_claim": ["automatic approval"],
  "answerable": true,
  "slices": ["policy", "cross-lingual", "table"]
}

Include answerable and unanswerable questions, ambiguous questions, outdated assumptions, access-denied documents, conflicting sources, exact identifiers, tables, and multi-hop comparisons.

Split data into development and locked test sets. Do not tune prompts or weights repeatedly against the final set. Refresh a production-representative shadow set on a schedule while keeping a stable core for trend analysis.

Label evidence consistently

Ask reviewers to judge passages as irrelevant, partially relevant, relevant, or essential. Provide written rules and overlap a sample between reviewers. Track agreement, adjudicate disagreements, and update unclear guidelines.

Passage labels become stale when chunking changes. Stable source spans—document ID, version, section, and offsets—survive chunker migrations better. At evaluation time, map retrieved chunks back to labeled spans.

Retrieval and ranking metrics

Use several metrics because each answers a different question:

Recall@K asks whether the necessary evidence appeared in the first K results.

Recall@K = relevant items retrieved in top K / all relevant items

Hit Rate@K asks whether at least one relevant result appeared. It is useful for single-fact questions but hides missing evidence in multi-document answers.

MRR rewards placing the first relevant result early:

MRR = mean(1 / rank of first relevant result)

nDCG@K handles graded relevance and rank position. Use it when “essential” evidence should count more than merely related evidence.

Also report zero-result rate, duplicate rate, context precision, context recall, and permission violations. One unauthorized result is a security incident, not a small metric decrease.

Answer and citation metrics

Measure distinct properties:

• correctness: does the answer match the reference or expert judgment?
• faithfulness/groundedness: is every claim supported by supplied evidence?
• completeness: are all required parts addressed?
• citation precision: do cited sources support their associated claims?
• citation recall/coverage: are claims that require evidence cited?
• abstention precision: when the system abstains, was the question truly unsupported?
• abstention recall: did it abstain on unsupported questions?
• style and policy compliance: did it follow required format and safety rules?

Exact match is appropriate for IDs or numbers but poor for prose. Semantic similarity alone can reward a fluent contradiction. Combine deterministic checks, span-level citation validation, expert review, and carefully calibrated model judges.

Use LLM judges as instruments, not authorities

A judge prompt should receive the question, approved reference criteria, generated answer, and cited evidence. Require structured scores and quoted supporting spans. Randomize answer order in pairwise tests and hide system names.

Validate every judge against a human-labeled sample. Report its agreement, false-positive pattern, model/version, and prompt version. A judge sharing the evaluated model’s biases can systematically miss errors.

Keep deterministic checks for numbers, dates, citations, forbidden claims, JSON shape, and source IDs. Escalate high-risk or low-confidence cases to humans.

Design experiments as ablations

Change one subsystem at a time:

1. lexical baseline
2. vector baseline
3. hybrid with RRF
4. hybrid plus rewrite
5. hybrid plus reranker
6. reranker plus context optimization

Record the complete configuration: corpus snapshot, parser/chunker/embedding versions, index parameters, query transforms, candidate counts, fusion weights, reranker, prompt, generator, and decoding settings.

Use paired comparisons because every configuration runs the same questions. Bootstrap confidence intervals for metric differences. For online binary outcomes, use a suitable proportion test; for ratings, use paired tests and report the distribution, not only an average.

Set guardrails before the experiment. For example: ship only if citation precision does not fall, p95 latency grows by less than 200 ms, and no protected language slice loses more than two recall points.

An illustrative results table

The numbers below demonstrate how to report an experiment; they are illustrative, not results from a real deployment.

Configuration	Recall@20	nDCG@10	Grounded answers	p95 latency	Cost/query
Vector only	0.71	0.58	72%	620 ms	$0.018
Hybrid + RRF	0.82	0.67	79%	690 ms	$0.019
+ safe rewrite	0.87	0.71	83%	810 ms	$0.020
+ cross-encoder	0.87	0.80	89%	1,060 ms	$0.024
+ evidence packing	0.87	0.81	93%	1,090 ms	$0.021

The correct interpretation is stage-specific: hybrid search and rewriting improved recall; reranking improved ordering; evidence packing improved grounding and reduced generation cost. A single “accuracy improved 21%” claim would obscure those mechanisms.

Slice before averaging

At minimum, report by:

• language and cross-language direction
• tenant or business domain
• query intent and complexity
• exact term versus semantic question
• source format: HTML, PDF, scan, table, slide, code
• document age and update frequency
• head versus tail traffic
• answerable versus deliberately unanswerable

Include sample count and confidence intervals. A 95% score on 20 French questions is not strong evidence of multilingual readiness.

Test the whole system under load

Quality tests do not expose queue saturation, cold caches, retry storms, or index contention. Load tests should replay realistic query and filter distributions while ingestion and compaction run.

Measure end-to-end and stage-level p50/p95/p99 latency, time to first token, throughput, timeout and partial-result rates, CPU/GPU/RAM, cache hit rate, tokens, and cost. Run steady-state, spike, soak, and dependency-degradation tests.

Test failure policy: if vector search times out, should lexical results continue? If reranking fails, can fused order be used? If the generator fails, can evidence links still be returned? Make degraded behavior explicit and observable.

Connect offline and online evaluation

Before release, require offline thresholds, security tests, and a performance budget. Then use shadow traffic, canaries, and A/B tests. Online metrics can include answer acceptance, citation opens, reformulation rate, escalation rate, task completion, time saved, latency, and cost.

User approval is not proof of correctness, and low clicks are not necessarily failure. Combine behavioral signals with sampled expert review and incident reports.

An evaluation system is successful when a team can answer: what changed, which cases improved, which regressed, whether the result is statistically and operationally meaningful, and how to roll it back.