Evaluate Retrieval and LLM Answers with Calibrated Rubrics

Build a layered evaluation

Start with a versioned question set. For each question, record acceptable evidence chunks, an answerability label and any required facts. Retrieval metrics can then be calculated without an LLM: recall at k asks whether relevant evidence was returned, while reciprocal rank rewards placing the first relevant item near the top.

Answer review needs a written rubric. A practical four-part rubric covers correctness, support by cited evidence, completeness and communication. Add a hard fail when the answer claims evidence that the citation does not contain. Reviewers should see the same question, evidence and answer in randomized order and have an “uncertain” path.

Use an LLM judge only after calibration

A judge model can help scale triage, but its scores are measurements from another fallible model. Calibrate it against blinded human labels, inspect disagreement by topic and language, and set a review band near the threshold. Recheck calibration after changing the judge model, prompt, rubric or answer system. Never let a judge evaluate its own response with no human control set.

Run retrieval metrics and judge calibration

Save as evaluate_rag.py. The judge scores are fixtures that make the calibration calculation reproducible. Replace them with raw, versioned judge outputs in a real evaluation.

python -m pip install scikit-learn
python evaluate_rag.py

# evaluate_rag.py
from statistics import mean
from sklearn.metrics import cohen_kappa_score, confusion_matrix

cases = [
    {"id": "q1", "relevant": {"refund#1"},
     "retrieved": ["support#1", "refund#1", "refund#2"],
     "human": 1, "judge_score": 0.91},
    {"id": "q2", "relevant": {"delivery#2"},
     "retrieved": ["delivery#2", "faq#4"],
     "human": 1, "judge_score": 0.67},
    {"id": "q3", "relevant": {"privacy#3"},
     "retrieved": ["terms#1", "faq#2"],
     "human": 0, "judge_score": 0.61},
    {"id": "q4", "relevant": {"billing#1"},
     "retrieved": ["billing#4", "billing#1"],
     "human": 0, "judge_score": 0.38},
    {"id": "q5", "relevant": {"account#2"},
     "retrieved": ["account#2"],
     "human": 1, "judge_score": 0.79},
]

def recall_at_k(case, k=3):
    return int(bool(case["relevant"] & set(case["retrieved"][:k])))

def reciprocal_rank(case):
    for rank, chunk_id in enumerate(case["retrieved"], 1):
        if chunk_id in case["relevant"]:
            return 1 / rank
    return 0.0

thresholds = [0.4, 0.5, 0.6, 0.7, 0.8]
best = None
for threshold in thresholds:
    predicted = [int(c["judge_score"] >= threshold) for c in cases]
    kappa = cohen_kappa_score([c["human"] for c in cases], predicted)
    candidate = (kappa, threshold, predicted)
    best = candidate if best is None or candidate[0] > best[0] else best

kappa, threshold, predicted = best
print("recall_at_3", mean(recall_at_k(c) for c in cases))
print("mrr", round(mean(reciprocal_rank(c) for c in cases), 3))
print("judge_threshold", threshold, "kappa", round(kappa, 3))
print("confusion_matrix", confusion_matrix(
    [c["human"] for c in cases], predicted).tolist())
print("manual_review", [c["id"] for c in cases
      if abs(c["judge_score"] - threshold) <= 0.1])

Expected output and interpretation

The script reports retrieval metrics, selects a threshold on the calibration fixtures, shows the confusion matrix and routes near-threshold cases to manual review. Five cases are far too few for a production claim. Use a larger held-out calibration set and report uncertainty.

A rubric reviewers can apply

Write examples at the boundary between pass and fail. Measure agreement between human reviewers before treating their label as ground truth. Resolve disagreements with documented adjudication, not a majority vote that hides rubric confusion.

Evaluation traps

• Testing only answerable questions inflates apparent usefulness.
• Reusing development cases for final reporting leaks tuning decisions into the test.
• Checking citation presence without citation support rewards decorative links.
• Accepting judge agreement on English as evidence for Urdu ignores a separate failure slice.
• Reporting one average hides poor performance on sensitive or rare questions.

Knowledge check

Evaluation references

• Hugging Face Evaluate
• Ragas metric concepts
• NIST Generative AI Profile
• Original RAG paper