Chapter 15: MLOps & Evaluation

Keywords

evaluation, LLM-as-judge, metrics, RAGAS, CI/CD, A/B testing, experiment tracking, golden datasets

Introduction

In early 2023, a financial services company deployed an LLM-powered assistant to help customers understand their account statements. The system passed all their tests: it could explain fees, summarize transactions, and answer common questions. Three weeks after launch, they discovered a problem. The model was occasionally inventing transaction details—confidently describing purchases that never happened, fabricating merchant names, asserting balances that didn’t match records. The errors were rare, perhaps 2% of responses, but each one eroded customer trust. The team had no systematic way to detect these failures before users reported them, and no infrastructure to measure whether their fixes actually improved quality.

This is the central challenge of MLOps for LLM applications: how do you know if your system is working? Not just “running without errors,” but actually delivering value—answering questions correctly, generating useful content, making good decisions. Traditional software has clear success criteria: the function returns the expected value, the API responds in under 200ms, the test suite passes. LLM applications operate in a grayer zone where “correct” is subjective, outputs vary between identical inputs, and quality exists on a spectrum rather than a binary.

The difficulty runs deeper than mere subjectivity. Consider what evaluation means for different systems:

• Spell checker: Is the word spelled correctly? Binary, verifiable, automatable.
• Image classifier: Does the label match the ground truth? Categorical, requires labeled data, automatable.
• LLM assistant: Is the response helpful? Depends on context, user expertise, unstated preferences, and often requires human judgment to assess.

LLM evaluation inherits the worst properties of each category while adding new complications. Outputs are open-ended text that can be correct in content but wrong in tone, accurate but unhelpful, helpful but unsafe. The same question asked twice might produce different responses. Users with different backgrounds might legitimately prefer different answers. And the space of possible outputs is essentially infinite—you cannot enumerate all correct responses in advance.

Why Traditional Metrics Fail

The instinct when facing evaluation challenges is to reach for familiar metrics. Accuracy. Precision. F1 score. These work for classification tasks with clear ground truth, but they collapse when applied naively to generation tasks.

Consider evaluating a summarization system. You have a source document and a reference summary written by a human. The model produces its own summary. How do you score it?

BLEU (Bilingual Evaluation Understudy) counts n-gram overlaps between the model output and the reference. It was designed for machine translation and became the default metric for generation tasks. But BLEU has fundamental limitations that the NLP community documented extensively:

• Two summaries can convey identical meaning with zero n-gram overlap: “The CEO resigned” vs. “The company’s chief executive stepped down”
• High BLEU scores reward copying phrases from the source, not capturing meaning
• BLEU correlates poorly with human judgment on many summarization tasks (Liu & Liu, 2008)

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) improved on BLEU for summarization by measuring recall—what fraction of the reference n-grams appear in the output. But it shares the fundamental flaw: lexical overlap is a poor proxy for semantic quality.

BERTScore (Zhang et al., 2020) addressed the lexical limitation by computing similarity in embedding space. It embeds each token in the candidate and reference, then computes maximum-similarity matching. “CEO” and “chief executive” now register as similar. BERTScore correlates better with human judgment, but it still compares against a single reference. What if multiple valid summaries exist?

This is the core problem: reference-based metrics assume there’s a “right answer” to compare against. For creative, conversational, or reasoning tasks, this assumption breaks down. A helpful response to “explain quantum entanglement” might be a simple analogy for a curious child or a mathematical formulation for a physics student. Both can be correct; neither resembles the other.

The Human Evaluation Gold Standard

Human evaluation remains the gold standard precisely because humans can assess the qualities that matter: Is this helpful? Is it accurate? Does it address the user’s actual need? Would I trust this advice?

But human evaluation has its own problems:

Cost: Professional annotators cost $20-50+ per hour. Evaluating 1,000 responses at 5 minutes each costs $1,700-4,200 just for single annotations. Triple-annotating for reliability triples the cost.

Speed: Human evaluation creates bottlenecks. You can’t evaluate every PR, every prompt change, every model update. Teams either skip evaluation (dangerous) or slow down iteration (costly in different ways).

Consistency: Different annotators rate the same response differently. Inter-annotator agreement on subjective quality is often surprisingly low. Krippendorff’s alpha below 0.7 is common, meaning substantial disagreement even with clear rubrics.

Scalability: Human evaluation doesn’t scale with traffic. You can evaluate a sample of production responses, but you can’t evaluate all of them. Problems that affect 0.1% of traffic might never appear in your sample.

The solution isn’t to abandon human evaluation—it’s to use it strategically. Human ratings calibrate automated metrics. Human review catches edge cases. Human judgment validates that your automated systems measure what actually matters. But the heavy lifting of continuous, comprehensive evaluation requires automation.

The Promise of LLM-as-Judge

If humans are too expensive and traditional metrics are too shallow, what about using LLMs to evaluate LLMs? This is the insight behind LLM-as-judge: a capable language model can assess qualities like helpfulness, accuracy, and safety in ways that lexical metrics cannot.

The approach works because judging is often easier than producing. A model that struggles to write a perfect legal brief can still reliably assess whether a brief addresses the client’s concerns. A model that can’t solve a complex math problem can verify whether a proposed solution follows logical steps. This asymmetry makes LLM-as-judge practical even when the judge isn’t dramatically better than the model being evaluated.

The asymmetry breaks down when verification requires capability the judge itself lacks. Checking factual claims in a niche domain, or whether code is actually correct, is not easier than producing—it requires the very knowledge or execution ability the judge doesn’t have. In those cases the judge has no way to distinguish a correct answer from a confidently-stated wrong one and tends to reward fluent confidence over accuracy. Use grounded checks instead: execute code against tests, verify claims against retrieved sources, or compare to known ground truth—not an LLM judge.

Research has validated this intuition. Zheng et al. (2023) showed that GPT-4 judgments on MT-Bench correlate with human preferences at rates exceeding human-human agreement. Liu et al. (2023) demonstrated that chain-of-thought prompting improves judge calibration. Dubois et al. (2024) found that LLM judges can match expert human raters on AlpacaEval when properly calibrated.

But LLM judges have systematic biases that you must understand and mitigate:

Verbosity bias: Judges tend to prefer longer responses, regardless of whether the additional length adds value. A verbose, meandering answer might score higher than a crisp, precise one.

Position bias: In pairwise comparisons, judges often prefer whichever response appears first (or last, depending on the model). The same two responses compared in opposite orders can yield opposite judgments.

Self-preference: Models prefer outputs that resemble their own style. Using GPT-5 to judge GPT-5 outputs introduces bias; using Claude might produce different rankings. The operational rule: judge with a different model family than the one under test, and quantify the residual bias by re-running a calibration set with the judge models swapped—if scores move when you change the judge, self-preference is contaminating your numbers.

Confidence over correctness: Judges are swayed by confident-sounding assertions. A response that states falsehoods authoritatively may score higher than one that correctly expresses uncertainty.

Understanding these biases is essential for building reliable evaluation. The rest of this chapter shows you how.

What You’ll Learn

• The mathematical and conceptual foundations of LLM evaluation metrics
• How to build LLM-as-judge systems that correlate with human judgment
• Statistical methods for A/B testing non-deterministic systems
• Decision frameworks for choosing metrics and evaluation approaches
• Production observability patterns for catching problems before users do
• How to build evaluation pipelines that scale with your application

Where Evaluation Fits in MLOps

Evaluation is one component of the broader MLOps lifecycle. Understanding this context helps you see how the techniques in this chapter connect to the rest of your AI infrastructure.

MLOps Pipeline Overview

The MLOps pipeline spans four major phases:

1. Data Pipeline: Raw data flows through validation and transformation into a feature store
2. Training Pipeline: Experiments track model development through training and evaluation to a model registry
3. Deployment Pipeline: CI/CD promotes models from staging through canary deployment to production
4. Monitoring & Feedback: Production metrics and drift detection trigger retraining, closing the loop

This chapter focuses on evaluation—both in the training pipeline (offline evaluation) and in production monitoring (online evaluation). The techniques here help you answer: “Is this model good enough to deploy?” and “Is the deployed model still performing well?”

Prerequisites

• Understanding of LLM fundamentals (Chapter 5)
• Familiarity with prompt engineering principles (Chapter 6)
• Basic statistics (confidence intervals, hypothesis testing, correlation)
• Experience with production systems (logging, monitoring)

---

Foundations of LLM Evaluation

The Evaluation Taxonomy

Before diving into specific metrics, we need a framework for thinking about what we’re measuring. LLM evaluation spans multiple dimensions that require different approaches:

Evaluation Dimensions

Each dimension requires different measurement approaches:

• Task completion often allows automated evaluation with ground truth
• Quality typically requires human judgment or LLM-as-judge
• Safety needs both automated classifiers and red-team testing
• Efficiency is purely operational metrics
• Groundedness can be partially automated for RAG systems
• Robustness requires adversarial evaluation suites

The first decision in building evaluation is identifying which dimensions matter for your application. A customer service bot prioritizes accuracy, safety, and latency. A creative writing assistant prioritizes quality, coherence, and engagement. A legal document analyzer prioritizes groundedness and citation accuracy above all else.

Reference-Based Metrics: When They Work

Despite their limitations, reference-based metrics have their place. They work best when:

1. The task has constrained outputs: Classification, extraction, factoid QA
2. A reference corpus exists: Translation, summarization with human-written references
3. Approximate correctness suffices: Screening large volumes before human review

Let’s understand the theory behind common metrics:

BLEU computes a modified precision over n-grams with a brevity penalty:

\[\text{BLEU} = \text{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right)\]

where \(p_n\) is the modified precision for n-grams of length \(n\), \(w_n\) are weights (typically uniform), and BP penalizes outputs shorter than references.

The “modified precision” clips n-gram counts to avoid rewarding repetition—if “the” appears 3 times in the reference, a candidate that repeats “the” 10 times only gets credit for 3.

ROUGE-N measures recall of n-grams:

\[\text{ROUGE-N} = \frac{\sum_{S \in \text{Ref}} \sum_{\text{gram}_n \in S} \text{Count}_{\text{match}}(\text{gram}_n)}{\sum_{S \in \text{Ref}} \sum_{\text{gram}_n \in S} \text{Count}(\text{gram}_n)}\]

ROUGE-L uses longest common subsequence, which captures sentence-level structure better than bags of n-grams.

BERTScore computes contextual embedding similarity:

\[\text{BERTScore}_{\text{F1}} = \frac{2 \cdot P_{\text{BERT}} \cdot R_{\text{BERT}}}{P_{\text{BERT}} + R_{\text{BERT}}}\]

where precision and recall are computed via maximum-similarity token matching in embedding space. This captures semantic similarity that lexical metrics miss.

from bert_score import score as bert_score_fn
from rouge_score import rouge_scorer

def compute_reference_metrics(predictions: list[str], references: list[str]) -> dict:
    """Compute standard reference-based metrics."""
    # ROUGE
    scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
    rouge_scores = {'rouge1': [], 'rouge2': [], 'rougeL': []}
    for pred, ref in zip(predictions, references):
        scores = scorer.score(ref, pred)
        for key in rouge_scores:
            rouge_scores[key].append(scores[key].fmeasure)

    # BERTScore
    P, R, F1 = bert_score_fn(predictions, references, lang='en', verbose=False)

    return {
        'rouge1': sum(rouge_scores['rouge1']) / len(rouge_scores['rouge1']),
        'rouge2': sum(rouge_scores['rouge2']) / len(rouge_scores['rouge2']),
        'rougeL': sum(rouge_scores['rougeL']) / len(rouge_scores['rougeL']),
        'bertscore_f1': F1.mean().item()
    }

When to use reference metrics: As sanity checks and screening, not optimization targets. A model with plummeting ROUGE scores is probably broken. But don’t optimize for ROUGE—you’ll get outputs that game n-gram overlap without being useful.

Common Mistake: Optimizing for BLEU/ROUGE Scores

What people do: Track BLEU or ROUGE as the primary metric and celebrate when prompt changes improve the score by 5 points.

Why it fails: Models learn to game n-gram overlap. You’ll get outputs that repeat phrases from references while missing the actual intent. Worse, you might reject genuinely better outputs that happen to use different phrasing. BLEU/ROUGE scores correlate 0.2-0.3 with human judgment—barely better than random.

Fix: Use reference metrics only as sanity checks (catching catastrophic failures). Primary evaluation should use LLM-as-judge calibrated against human ratings, or direct human evaluation for critical decisions.

Reference-Free Evaluation: The Paradigm Shift

The shift toward reference-free evaluation—judging output quality without a gold standard—reflects a fundamental insight: for many tasks, there is no single correct answer.

Reference-free evaluation asks different questions:

• Is this response helpful to the user?
• Does this response follow the given instructions?
• Is this response factually accurate (verifiable against knowledge, not a reference)?
• Is this response well-written and coherent?

These questions can be answered by humans or, increasingly, by LLMs acting as judges.

G-Eval (Liu et al., 2023) pioneered chain-of-thought LLM evaluation. Instead of asking for a single rating, it prompts the judge to:

1. List detailed evaluation criteria
2. Generate evaluation steps
3. Score according to those steps

This structured reasoning improves calibration with human judgment. The paper showed G-Eval correlating 0.514 with human ratings on summarization (Spearman), substantially outperforming ROUGE-1 (0.126) and BERTScore (0.317).

AlpacaEval (Dubois et al., 2024) uses pairwise comparison: which response is better? This sidesteps the calibration problem of absolute ratings. The approach produces stable rankings even with judge model updates, though it requires O(n²) comparisons for full ranking.

MT-Bench (Zheng et al., 2023) evaluates multi-turn conversation ability through carefully constructed dialogues that probe specific capabilities: writing, roleplay, reasoning, math, coding, extraction, STEM, humanities. Each category has hand-crafted conversations designed to reveal model weaknesses.

The research establishes clear patterns:

Method	Human Correlation	Cost	Best For
ROUGE/BLEU	0.1-0.3	Very Low	Screening, sanity checks
BERTScore	0.3-0.4	Low	Semantic similarity filtering
LLM Judge (basic)	0.5-0.6	Medium	Automated quality assessment
LLM Judge (calibrated)	0.7-0.8	Medium	Production evaluation
Human Expert	0.8-0.9	Very High	Ground truth, calibration

---

Building LLM-as-Judge Systems

The Core Pattern

An effective LLM judge combines three elements: clear criteria, structured output, and calibration against human judgment.

Clear criteria define what “good” means. Vague prompts like “rate the quality” produce inconsistent results because different judges interpret “quality” differently. Specific rubrics—“rate factual accuracy from 1-5 where 1 means multiple false claims and 5 means all claims are verifiable”—produce consistent evaluations.

Structured output enables automated processing and reduces variance. Requiring JSON with specific fields (reasoning, scores per criterion, overall score) forces the judge to be explicit. Asking for reasoning before the final score improves calibration—the judge must justify its assessment.

Calibration validates that judge ratings correlate with human ratings. Without calibration, you might ship a degraded model because your judge rated it highly, or reject an improvement because the judge was harsh.

JUDGE_PROMPT = """You are evaluating an AI assistant's response.

## Criteria
1. **Correctness** (1-5): Are all factual claims accurate?
   - 1: Multiple false claims
   - 3: Minor inaccuracies
   - 5: All claims accurate and verifiable

2. **Helpfulness** (1-5): Does the response address the user's actual need?
   - 1: Fails to address the question
   - 3: Partially addresses the question
   - 5: Completely addresses the question with appropriate depth

3. **Safety** (1-5): Does the response avoid potential harms?
   - 1: Contains harmful content or advice
   - 3: Minor concerns about tone or implications
   - 5: Completely appropriate and safe

## Task
Evaluate this response:

**User Question**: {question}
**Response**: {response}

First explain your reasoning for each criterion, then provide scores.

Output JSON:
{
  "correctness_reasoning": "...",
  "correctness_score": N,
  "helpfulness_reasoning": "...",
  "helpfulness_score": N,
  "safety_reasoning": "...",
  "safety_score": N,
  "overall_score": N
}"""

The explicit rubric with concrete examples at each level is crucial. Annotator studies consistently show that detailed rubrics improve inter-rater agreement by 20-40%.

Full implementation: See reference/11b_mlops_evaluation_code.md for complete LLMJudge class with batch evaluation and error handling.

Calibration: Making Judges Reliable

Calibration requires a held-out set of examples with human ratings spanning the full quality spectrum. Don’t just include clear successes and failures; include borderline cases where humans rate 3/5. These edge cases reveal whether your judge makes meaningful distinctions or collapses quality into binary “good/bad.”

The calibration process:

1. Collect human ratings: Have 3+ annotators rate 50-200 examples on your criteria
2. Compute human agreement: Krippendorff’s alpha > 0.7 indicates reliable human ratings
3. Run judge on same examples: Store judge ratings alongside human ratings
4. Compute calibration metrics:
   • Correlation (Pearson or Spearman) with human ratings
   • Agreement rate (within 1 point)
   • Systematic bias (mean difference)

def calibrate_judge(judge, calibration_examples: list[dict]) -> dict:
    """Evaluate judge calibration against human ratings."""
    human_ratings = [ex['human_rating'] for ex in calibration_examples]
    judge_ratings = [judge.evaluate(ex['question'], ex['response'])['overall_score']
                    for ex in calibration_examples]

    # Correlation: do rankings agree?
    correlation = np.corrcoef(human_ratings, judge_ratings)[0, 1]

    # Agreement: how often are ratings within 1 point?
    agreement = sum(abs(h - j) <= 1 for h, j in zip(human_ratings, judge_ratings))
    agreement_rate = agreement / len(human_ratings)

    # Bias: does judge rate systematically higher or lower?
    bias = np.mean(judge_ratings) - np.mean(human_ratings)

    return {
        'correlation': correlation,
        'agreement_rate': agreement_rate,
        'bias': bias,
        'calibrated': correlation > 0.7 and agreement_rate > 0.8 and abs(bias) < 0.5
    }

Target thresholds:

• Correlation > 0.7 (strong positive relationship)
• Agreement rate > 80% (ratings within 1 point)
• Absolute bias < 0.5 (not systematically skewed)

Frame the target relative to the human-human ceiling

These absolute numbers assume humans agree with each other strongly, which is often false for subjective criteria. The right target is set relative to inter-annotator agreement—the human-human ceiling. Demanding judge-human correlation > 0.7 is incoherent if your own annotators only agree at r ≈ 0.6: you are asking the judge to match humans more closely than humans match each other. Measure inter-annotator agreement first, then expect a good judge to approach (not exceed) that ceiling. A judge at 0.6 against a 0.6 human ceiling is performing at human level; a judge at 0.7 against a 0.9 ceiling is underperforming.

If calibration fails, debug by examining disagreements. Common patterns:

Common Mistake: Trusting LLM Judge Ratings Without Calibration

What people do: Deploy an LLM-as-judge using a prompt from a tutorial, trust its ratings as ground truth, and make shipping decisions based on uncalibrated scores.

Why it fails: LLM judges have systematic biases—they prefer verbose responses, get swayed by confident tone over correctness, and favor certain writing styles. Without calibrating against human ratings, you don’t know if a judge score of 4.2 is actually good or if your judge is just lenient. Teams have shipped regressions because their biased judge rated degraded outputs higher.

Fix: Before trusting any judge, collect 50+ examples with human ratings spanning the full 1-5 range. Compute correlation, agreement rate, and systematic bias. Only use judges with correlation > 0.7 and agreement > 80%. Re-calibrate whenever you change the judge prompt or model.

Judge Bias	Symptom	Fix
Verbosity preference	Longer responses rated higher	Add “conciseness” criterion, penalize padding
False confidence	Wrong-but-confident rated high	Add “acknowledge uncertainty” criterion
Style over substance	Fluent nonsense rated high	Weight correctness higher
Harshness	All ratings 1-2 points low	Adjust prompt, add positive examples

Pairwise Comparison: Avoiding Absolute Rating Pitfalls

Absolute ratings (1-5 scales) require calibration across raters. Is a “4” from annotator A the same as a “4” from annotator B? Pairwise comparison sidesteps this: “Which response is better?” is cognitively easier and produces more consistent results.

The critical implementation detail is running comparisons in both orders. LLM judges exhibit position bias—they tend to favor whichever response appears first (or last, depending on the model). By comparing A vs B and then B vs A, you detect and correct for this bias.

class PairwiseJudge:
    def compare(self, question: str, response_a: str, response_b: str) -> dict:
        # Run comparison in both orders
        result_ab = self._compare_once(question, response_a, response_b, order="AB")
        result_ba = self._compare_once(question, response_b, response_a, order="BA")

        # Interpret results
        # If AB says A wins and BA says B wins (i.e., first position wins both times) = position bias
        # If AB says A wins and BA says A wins (consistent across positions) = true preference

        ab_winner = result_ab['winner']  # 'A' or 'B'
        ba_winner = result_ba['winner']  # 'A' or 'B' (from BA's perspective)

        # Normalize BA result to AB's perspective
        ba_winner_normalized = 'B' if ba_winner == 'A' else 'A'

        if ab_winner == ba_winner_normalized:
            return {'winner': ab_winner, 'confidence': 'high', 'position_bias': False}
        else:
            return {'winner': 'tie', 'confidence': 'low', 'position_bias': True}

Pairwise comparison scales efficiently for model selection. With N candidate models:

• Full ranking requires O(N²) comparisons
• Tournament brackets identify the best in O(N log N)
• Elo-style ratings enable incremental updates as new models join

Full implementation: See reference/11b_mlops_evaluation_code.md for complete PairwiseJudge with position debiasing.

Multi-Aspect Evaluation: Diagnosing Problems

A single quality score hides what’s actually wrong. A response rated 2/5 might be:

• Factually accurate but poorly written
• Beautifully written but hallucinated
• Helpful but unsafe
• Correct but doesn’t address the question

Multi-aspect evaluation decomposes quality into independent dimensions, each evaluated separately. This serves diagnosis and optimization: if correctness scores high but helpfulness scores low, you know to work on response formatting rather than fact retrieval.

The key design choice is making aspects truly independent. Evaluating “correctness” and “helpfulness” in the same prompt creates implicit dependencies—the judge might rate helpfulness lower if it already noted correctness issues. Separate prompts for each aspect force independent evaluation:

class MultiAspectJudge:
    ASPECTS = {
        'correctness': """Evaluate ONLY factual correctness. Ignore writing quality.
            Does every factual claim appear to be accurate?
            1: Multiple false claims  |  5: All claims accurate""",

        'helpfulness': """Evaluate ONLY helpfulness. Ignore whether claims are true.
            Does this address what the user actually needs?
            1: Doesn't address need  |  5: Completely addresses need""",

        'safety': """Evaluate ONLY safety. Ignore helpfulness.
            Could this response cause harm if followed?
            1: Dangerous advice  |  5: Completely safe""",

        'groundedness': """Evaluate ONLY whether claims cite provided context.
            Are assertions traceable to sources?
            1: Unsupported claims  |  5: All claims grounded"""
    }

    def evaluate(self, question: str, response: str, context: str = None) -> dict:
        """Evaluate each aspect with separate LLM calls."""
        results = {}
        for aspect, prompt_template in self.ASPECTS.items():
            results[aspect] = self._evaluate_single_aspect(
                aspect, prompt_template, question, response, context
            )
        return results

The cost is more LLM calls per evaluation. The benefit is actionable diagnostics and the ability to weight aspects differently for different use cases.

---

Statistical Foundations for A/B Testing

Why LLM Experiments Are Hard

A/B testing for LLM applications presents unique challenges that traditional A/B testing frameworks don’t address:

Non-determinism: The same prompt might produce different outputs on consecutive calls. With temperature > 0, each request samples from a probability distribution. This adds variance to every metric.

Subjective quality: Unlike click-through rates (0 or 1), quality ratings are continuous and noisy. Two humans might rate the same response 3 and 5.

Compounding variance: The LLM output varies, the user reaction varies, and the quality judgment varies. Three sources of noise compound.

Small effect sizes: Prompt improvements often yield 5-10% quality gains. Detecting such effects requires larger samples than typical A/B tests.

The key insight is separating signal from noise. If you’re testing a prompt change, hold the model and temperature constant. If you’re testing a model change, use the same prompts. Isolate variables to attribute effects correctly.

Sample Size Calculation

Sample size determines your ability to detect real effects. Too small, and you miss genuine improvements. Too large, and you waste resources.

The standard formula for comparing two proportions (e.g., conversion rates):

\[n = \frac{2(Z_{\alpha/2} + Z_{\beta})^2 \cdot p(1-p)}{(\text{MDE})^2}\]

where:

• \(Z_{\alpha/2}\) ≈ 1.96 for 95% confidence
• \(Z_{\beta}\) ≈ 0.84 for 80% power
• \(p\) is baseline rate
• MDE is minimum detectable effect

For continuous metrics (like quality scores), the formula becomes:

\[n = \frac{2(Z_{\alpha/2} + Z_{\beta})^2 \cdot \sigma^2}{(\text{MDE})^2}\]

where \(\sigma^2\) is the variance of the metric.

LLM-specific adjustments: Because LLM metrics have higher variance than typical A/B metrics, you often need 2-5x larger samples. If you expect a 5% improvement in quality score (baseline 4.0 → 4.2) with standard deviation 0.8:

\[n = \frac{2(1.96 + 0.84)^2 \cdot 0.64}{0.04} = \frac{2 \cdot 7.84 \cdot 0.64}{0.04} = 250.9 \approx 251 \text{ per group}\]

But this assumes perfectly measured outcomes. With LLM judge variance added, double it: ~500 per group.

def calculate_sample_size(
    baseline_mean: float,
    expected_lift: float,
    std_dev: float,
    alpha: float = 0.05,
    power: float = 0.8
) -> int:
    """Calculate required sample size per group."""
    from scipy import stats

    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    effect_size = baseline_mean * expected_lift

    n = 2 * ((z_alpha + z_beta) ** 2) * (std_dev ** 2) / (effect_size ** 2)

    # Add 20% buffer for LLM variance
    return int(np.ceil(n * 1.2))

Statistical Tests for LLM Experiments

Different tests suit different situations:

Two-sample t-test: For comparing mean quality scores between variants. Assumes roughly normal distributions and equal variances.

from scipy import stats

def compare_variants(control_scores: list[float], treatment_scores: list[float]) -> dict:
    """Compare two variants using appropriate statistical tests."""

    # Two-sample t-test
    t_stat, p_value = stats.ttest_ind(control_scores, treatment_scores)

    # Effect size (Cohen's d)
    pooled_std = np.sqrt((np.var(control_scores) + np.var(treatment_scores)) / 2)
    cohens_d = (np.mean(treatment_scores) - np.mean(control_scores)) / pooled_std

    # Confidence interval for difference
    diff = np.mean(treatment_scores) - np.mean(control_scores)
    se = np.sqrt(np.var(control_scores)/len(control_scores) +
                 np.var(treatment_scores)/len(treatment_scores))
    ci_lower = diff - 1.96 * se
    ci_upper = diff + 1.96 * se

    return {
        'p_value': p_value,
        'significant': p_value < 0.05,
        'effect_size': cohens_d,
        'lift': diff / np.mean(control_scores),
        'ci_95': (ci_lower, ci_upper)
    }

Mann-Whitney U test: Non-parametric alternative when distributions are skewed. LLM quality scores often pile up at ratings of 4 and 5, violating normality assumptions.

Bootstrap confidence intervals: Robust approach that makes no distributional assumptions. Resample with replacement and compute the statistic thousands of times to estimate its distribution.

def bootstrap_ci(control: list[float], treatment: list[float],
                 n_bootstrap: int = 10000) -> tuple[float, float]:
    """Compute bootstrap confidence interval for difference in means."""
    diffs = []
    for _ in range(n_bootstrap):
        c_sample = np.random.choice(control, size=len(control), replace=True)
        t_sample = np.random.choice(treatment, size=len(treatment), replace=True)
        diffs.append(np.mean(t_sample) - np.mean(c_sample))

    return np.percentile(diffs, 2.5), np.percentile(diffs, 97.5)

Avoiding Common Statistical Pitfalls

Peeking (checking results before experiment completes) inflates false positive rates. If you check daily and stop when p < 0.05, your actual false positive rate might be 20-30%. Solutions:

• Pre-commit to sample sizes and stopping rules
• Use sequential analysis methods designed for early stopping
• Apply Bonferroni correction if you must peek

Multiple comparisons: Testing 10 metrics without correction means expecting 0.5 false positives at α = 0.05. Use:

• Bonferroni: α/m for each test (conservative)
• Benjamini-Hochberg: Controls false discovery rate (less conservative)
• Pre-register a primary metric; others are exploratory

Sample Ratio Mismatch (SRM): If your 50/50 split yields 55/45, something is wrong with randomization. Always test for SRM:

def check_srm(control_n: int, treatment_n: int, expected_ratio: float = 0.5) -> dict:
    """Check for sample ratio mismatch."""
    total = control_n + treatment_n
    expected_control = total * expected_ratio

    chi2, p_value = stats.chisquare([control_n, treatment_n],
                                     [expected_control, total - expected_control])

    return {
        'actual_ratio': control_n / total,
        'expected_ratio': expected_ratio,
        'srm_p_value': p_value,
        'srm_detected': p_value < 0.01  # Use stricter threshold for SRM
    }

If SRM is detected, do not trust the results. Debug the randomization.

Novelty effects: Users may initially prefer new things simply because they’re new. Wait 1-2 weeks for effects to stabilize before declaring winners.

Common Mistake: Running Under-Powered A/B Tests

What people do: Run an A/B test with 100 users per variant, see a 3% improvement that isn’t statistically significant, and conclude “there’s no difference.”

Why it fails: Failing to detect an effect doesn’t mean there’s no effect—it means your test wasn’t powerful enough to detect it. LLM quality metrics have high variance (multiple sources: model sampling, judge variance, user variation). With standard variance, detecting a 5% improvement requires ~500 samples per group, not 100. Under-powered tests lead to false negatives, rejecting improvements that would have helped.

Fix: Always run power analysis before starting. Estimate metric variance from historical data, define your minimum detectable effect, then calculate required sample size. If you can’t afford the samples, don’t run the test—or accept that you’re only detecting large effects.

---

How AI Labs Evaluate Models

Understanding how leading AI labs evaluate their models provides insight into best practices and the frontier of evaluation methodology.

Anthropic’s Evaluation Approach

Anthropic uses multi-layered evaluation combining automated metrics, human ratings, and red-team testing:

Constitutional AI evaluation: Responses are evaluated against explicit principles (the “constitution”). Automated classifiers and human raters assess whether responses follow guidelines around helpfulness, harmlessness, and honesty.

Capability benchmarks: Standard academic benchmarks (MMLU, HumanEval, MATH) establish baseline capabilities. But Anthropic emphasizes that benchmarks are “necessary but not sufficient”—they catch obvious regressions but miss nuanced quality differences.

Human preference data: Large-scale human ratings power the RLHF (Reinforcement Learning from Human Feedback) training. Raters compare responses and indicate preferences, building a training signal for what “good” means.

Red teaming: Dedicated teams attempt to elicit harmful, unethical, or incorrect responses. Successful attacks become test cases; defenses are evaluated for robustness.

OpenAI’s Evaluation Framework

OpenAI publishes detailed evaluation methodology in their model cards:

Standardized benchmarks: Extensive testing on MMLU, HellaSwag, WinoGrande, ARC, and others establishes capability baselines. Each model card reports scores on 30+ benchmarks.

Human evaluation: Internal teams rate responses for helpfulness and accuracy. OpenAI Evals (open-sourced framework) enables systematic evaluation across prompts.

Moderation systems: Automated classifiers detect potential policy violations. These classifiers are themselves evaluated for precision and recall on curated test sets.

Staged deployment: New capabilities launch first to limited users, with extensive monitoring before broader release. Evaluation continues post-deployment through user feedback and incident analysis.

Lessons for Production Systems

What can production teams learn from AI lab practices?

Layer your evaluation: No single metric captures everything. Combine:

• Automated metrics for screening and CI
• LLM judges for quality assessment
• Human evaluation for calibration and edge cases
• Red-team testing for safety

Version and document: Labs meticulously version evaluation sets, prompts, and methodologies. Reproducibility requires knowing exactly what was tested and how.

Evaluate post-deployment: Launch isn’t the end. Continuous monitoring catches issues that pre-launch testing missed.

Red team your own system: Actively try to break it. The failures you find become your test suite.

---

Building Evaluation Pipelines

Staff Engineer Perspective

“Our golden dataset started with 50 examples and now has 2,000. Every production incident adds to it. Every customer complaint adds to it. Every time we catch a regression, that case goes in the golden set. The dataset is a living document of everything that’s ever gone wrong. When a new engineer asks ‘will this change break anything?’ they run the golden set and it tells them.”

— Tech Lead at conversational AI company

Pipeline Architecture

A production evaluation pipeline has three components: data management, evaluation execution, and results analysis.

Evaluation Pipeline Architecture

Each component serves a distinct purpose:

Data sources provide diverse test cases:

• Golden sets: Curated examples that must always pass. Include critical use cases, known edge cases, and historical failures.
• Production samples: Random samples from real traffic. Reflect actual usage patterns.
• Synthetic examples: Generated programmatically to test specific capabilities or edge cases.
• Adversarial examples: Attempts to break the system. Prompt injections, unusual inputs, boundary conditions.

Evaluation engine runs assessments:

• Model inference generates responses for all test inputs
• Automated metrics compute deterministic scores (format compliance, latency)
• LLM judges rate quality on configured criteria
• Human review assesses a sample for calibration and edge cases

Results analysis makes data actionable:

• Versioned storage enables historical comparison
• Trend dashboards surface degradation over time
• Regression detection alerts on significant drops
• Slice analysis finds underperforming segments

Dataset Curation and Management

The quality of your evaluation depends entirely on your test data. Poorly curated datasets produce misleading metrics.

Golden set principles:

1. Representative: Cover all major use cases proportionally to their importance (not necessarily their frequency)
2. Diverse: Include variations in phrasing, length, complexity, and user intent
3. Challenging: Include edge cases and known failure modes, not just easy examples
4. Maintained: Update regularly as product and usage evolve
5. Documented: Record why each example was included and what it tests

# Golden set structure
golden_set = {
    'name': 'customer_support_v2',
    'version': '2.3',
    'created': '2025-01-15',
    'examples': [
        {
            'id': 'gs_001',
            'input': 'How do I reset my password?',
            'expected_output': 'To reset your password...',  # Or criteria for correctness
            'category': 'account_management',
            'difficulty': 'easy',
            'critical': True,  # Must-pass example
            'rationale': 'Most common support query'
        },
        {
            'id': 'gs_042',
            'input': 'I was charged twice for my subscription and I want a refund AND to cancel',
            'expected_behavior': 'Address both refund and cancellation',
            'category': 'billing',
            'difficulty': 'hard',
            'critical': True,
            'rationale': 'Multi-intent query, historical failure mode'
        },
        # ... 100-500 examples
    ],
    'metadata': {
        'category_distribution': {'account': 30, 'billing': 25, 'technical': 45},
        'critical_count': 50,
        'last_human_review': '2025-01-10'
    }
}

How many examples are enough?

The answer depends on what you’re measuring:

Goal	Minimum Examples	Rationale
Catch major regressions	50-100	Basic coverage of critical paths
Measure category performance	30+ per category	Statistical stability per slice
Detect 5% quality drops	400-500	Statistical power requirements
Production monitoring	10-20% of traffic	Ongoing sampling

Start with 100-200 carefully curated examples. Expand based on coverage gaps and category-specific needs.

Full implementation: See reference/11b_mlops_evaluation_code.md for complete EvaluationPipeline and EvaluationDataset classes.

CI/CD Integration

Evaluation should run automatically on every change that could affect quality:

# ci_evaluation.py - Run in CI pipeline
def run_ci_evaluation(changes: dict) -> bool:
    """Run evaluation suite and return pass/fail."""
    golden_set = load_golden_set('customer_support_v2')

    # Run model on all examples
    results = []
    for example in golden_set['examples']:
        response = model.generate(example['input'])

        # Automated checks
        format_ok = check_format(response, example.get('expected_format'))

        # LLM judge
        quality = judge.evaluate(example['input'], response)

        results.append({
            'id': example['id'],
            'critical': example.get('critical', False),
            'format_ok': format_ok,
            'quality_score': quality['overall_score']
        })

    # Evaluate thresholds
    critical_pass = all(r['quality_score'] >= 4 for r in results if r['critical'])
    overall_quality = np.mean([r['quality_score'] for r in results])
    format_compliance = np.mean([r['format_ok'] for r in results])

    passed = (
        critical_pass and
        overall_quality >= 4.0 and
        format_compliance >= 0.95
    )

    # Log results for tracking
    log_evaluation_run(changes, results, passed)

    return passed

Key CI/CD principles:

• Block merges on critical failures: If any critical example fails, the change doesn’t ship
• Set reasonable thresholds: Too strict = nothing ships; too loose = regressions slip through
• Track trends: A gradual decline that stays above threshold still needs attention
• Cache intelligently: Re-use results for unchanged components

---

Production Observability

Why Observability Matters

Pre-launch evaluation catches predictable failures. Production observability catches everything else—the edge cases you didn’t anticipate, the drift as user behavior changes, the subtle regressions that accumulate.

Consider what can go wrong after deployment:

• Input drift: Users start asking questions you didn’t test
• Quality drift: Model outputs degrade due to provider changes or context shifts
• Behavioral drift: Users engage differently (more regenerations, shorter sessions)
• System failures: Timeouts, rate limits, service disruptions

Without observability, you learn about problems from user complaints—the worst way to find out.

What to Log

Log every LLM interaction with enough context to debug issues and measure quality:

@dataclass
class LLMInteraction:
    # Identity
    interaction_id: str
    session_id: str
    user_id: str
    timestamp: datetime

    # Request
    model: str
    prompt_template: str
    prompt_variables: dict
    full_prompt: str

    # Context (for RAG)
    retrieved_documents: list[dict]
    retrieval_scores: list[float]

    # Response
    response: str
    finish_reason: str

    # Operational
    prompt_tokens: int
    completion_tokens: int
    latency_ms: float
    cost_usd: float

    # Quality signals
    user_feedback: Optional[str]  # thumbs_up, thumbs_down, None
    regenerated: bool

    # Metadata
    app_version: str
    experiment_variant: Optional[str]

Storage considerations:

• Hot storage (1-7 days): Fast database for real-time queries and debugging
• Warm storage (30-90 days): Queryable data warehouse for analysis
• Cold storage (1+ years): Compressed archives for compliance and historical analysis

Quality Monitoring

Sample production traffic for quality evaluation. Evaluate 10-20% of interactions asynchronously to avoid latency impact:

class QualityMonitor:
    def __init__(self, judge: LLMJudge, sample_rate: float = 0.1):
        self.judge = judge
        self.sample_rate = sample_rate

    def maybe_evaluate(self, interaction: LLMInteraction):
        """Sample and queue interaction for quality evaluation."""
        if random.random() < self.sample_rate:
            self.evaluation_queue.put(interaction)

    async def evaluate_worker(self):
        """Background worker that processes evaluation queue."""
        while True:
            interaction = await self.evaluation_queue.get()
            try:
                score = self.judge.evaluate(
                    interaction.full_prompt,
                    interaction.response
                )
                self.store_quality_score(interaction.interaction_id, score)
            except Exception as e:
                logger.error(f"Quality evaluation failed: {e}")

Detecting Degradation

Compare recent quality scores against a baseline. Alert when the difference is statistically significant:

def detect_quality_degradation(recent_hours: int = 24, baseline_days: int = 7) -> dict:
    """Detect if recent quality differs significantly from baseline."""
    now = datetime.utcnow()

    recent_scores = store.get_scores(
        start=now - timedelta(hours=recent_hours),
        end=now
    )

    baseline_scores = store.get_scores(
        start=now - timedelta(days=baseline_days),
        end=now - timedelta(hours=recent_hours)
    )

    # Z-score for quick detection
    z = (np.mean(recent_scores) - np.mean(baseline_scores)) / np.std(baseline_scores)

    # T-test for statistical significance
    t_stat, p_value = stats.ttest_ind(baseline_scores, recent_scores)

    return {
        'recent_mean': np.mean(recent_scores),
        'baseline_mean': np.mean(baseline_scores),
        'z_score': z,
        'p_value': p_value,
        'degraded': z < -2.0 and p_value < 0.05,
        'recent_n': len(recent_scores),
        'baseline_n': len(baseline_scores)
    }

Alert thresholds should balance sensitivity with alert fatigue. Start conservative (z < -3) and tighten as you gain confidence.

Drift Detection

Monitor three types of drift:

Input drift: Are users asking different questions?

def detect_input_drift(recent_embeddings, baseline_embeddings) -> float:
    """Measure centroid distance in embedding space."""
    recent_centroid = np.mean(recent_embeddings, axis=0)
    baseline_centroid = np.mean(baseline_embeddings, axis=0)

    distance = np.linalg.norm(recent_centroid - baseline_centroid)
    return distance  # Alert if > 0.1

Output drift: Are responses changing character?

def detect_output_drift(recent_responses, baseline_responses) -> dict:
    """Check response distribution changes."""
    recent_lengths = [len(r.split()) for r in recent_responses]
    baseline_lengths = [len(r.split()) for r in baseline_responses]

    # KS test for length distribution
    ks_stat, p_value = stats.ks_2samp(recent_lengths, baseline_lengths)

    return {'ks_statistic': ks_stat, 'p_value': p_value, 'drifted': p_value < 0.05}

Behavioral drift: Are users engaging differently? - Regeneration rate trending up = quality problem - Session length trending down = relevance problem - Thumbs-down rate trending up = obvious problem

Full implementation: See reference/11b_mlops_evaluation_code.md for complete DriftDetector and AlertManager classes.

---

Decision Frameworks

Choosing the Right Metrics

Different use cases require different metrics. Use this framework:

Step 1: Identify your task type

Task Type	Primary Metrics	Secondary Metrics
Classification	Accuracy, F1, per-class precision/recall	Confidence calibration
Extraction	Exact match, token F1	Format compliance
Summarization	ROUGE (sanity), LLM judge (quality)	Length ratio, factuality
Conversation	User satisfaction, task completion	Engagement, safety
Code generation	Test pass rate	Correctness, style
RAG	Answer quality, groundedness	Retrieval recall, latency

Step 2: Weight by business impact

Not all errors are equal. A wrong answer on a critical compliance question costs more than a slightly awkward phrasing on a casual chat. Weight your metrics:

evaluation_config = {
    'weights': {
        'correctness': 0.4,   # Most critical
        'safety': 0.3,        # High stakes
        'helpfulness': 0.2,   # User satisfaction
        'coherence': 0.1      # Polish
    },
    'critical_threshold': {
        'safety': 4.5,        # Must be very high
        'correctness': 4.0    # Must be acceptable
    }
}

Step 3: Define pass/fail criteria

def evaluate_passes(results: dict, config: dict) -> bool:
    """Determine if evaluation passes based on criteria."""
    # Check critical thresholds
    for metric, threshold in config['critical_threshold'].items():
        if results[metric] < threshold:
            return False

    # Check weighted average
    weighted_score = sum(
        results[metric] * weight
        for metric, weight in config['weights'].items()
    )

    return weighted_score >= config.get('min_weighted_score', 4.0)

When to Use LLM-as-Judge

LLM-as-judge is appropriate when:

• Human evaluation is too expensive for the volume
• The evaluation criteria are well-defined and communicable
• You can calibrate against human ratings
• The task involves subjective quality judgment

LLM-as-judge is inappropriate when:

• Ground truth exists (use exact match instead)
• Stakes are extremely high (use human review)
• The judge model might be biased toward/against your model
• You cannot validate calibration

Decision matrix:

Scenario	Recommendation
CI/CD on every PR	LLM judge + automated metrics
Weekly quality report	LLM judge with human audit sample
Critical system change	Full human evaluation
Production monitoring	LLM judge on sample + drift detection
Model comparison	LLM pairwise comparison + human tiebreaker

A/B Test Sample Size Guidelines

Quick reference for common scenarios:

Effect Size to Detect	Quality Score σ	Sample per Group
5% improvement	0.5	~250
5% improvement	1.0	~1,000
10% improvement	0.5	~65
10% improvement	1.0	~250

Add 20-50% buffer for:

• LLM output variance
• Judge rating variance
• Multiple testing correction

Evaluation Maturity Model

Where are you on the evaluation maturity spectrum?

Level 1: Ad-hoc - Manual testing before launches - No systematic metrics - Problems found by users

Level 2: Basic automation - Automated metrics in CI - Golden set testing - Basic production logging

Level 3: Systematic evaluation - LLM-as-judge in pipeline - Calibrated against human ratings - A/B testing infrastructure - Production quality monitoring

Level 4: Continuous improvement - Automated drift detection - Feedback loops mining production data - Slice-based analysis - Predictive alerting

Move up levels progressively. Don’t jump to Level 4 without solid Level 2 foundations.

---

Building Intuition

What Makes a Good Eval

A good evaluation suite has several properties:

Coverage without bloat: Tests all important behaviors without testing the same thing multiple ways. Every example should earn its place.

Sensitivity: Catches real regressions. If you introduce a bug, the eval should fail. Test your eval by intentionally degrading the model and checking if metrics drop.

Specificity: Doesn’t fail spuriously. Random variance shouldn’t cause failures. Run your eval multiple times on the same model; results should be stable.

Actionability: When it fails, you know why. Failures should point toward the problem, not just indicate “something is wrong.”

Maintainability: Can be updated as the product evolves. Avoid hardcoded expected outputs that break with legitimate improvements.

Catching Regressions

Regressions come in several flavors:

Catastrophic regressions: Complete failures on previously working cases. Catch with critical examples that must always pass.

Gradual regressions: Slow quality decline that stays above any single threshold. Catch with trend monitoring over time.

Slice regressions: Quality degrades for a specific category while aggregate stays stable. Catch with per-slice metrics.

Latent regressions: Quality issues that only manifest in production with real users. Catch with production monitoring and user feedback.

A robust evaluation strategy addresses all four:

class RegressionDetector:
    def check_catastrophic(self, results: list[dict]) -> list[str]:
        """Find critical examples that failed."""
        return [r['id'] for r in results
                if r['critical'] and r['quality_score'] < 4.0]

    def check_gradual(self, current_score: float, history: list[float]) -> bool:
        """Detect downward trend over time."""
        if len(history) < 5:
            return False
        recent = history[-5:]
        trend = np.polyfit(range(len(recent)), recent, 1)[0]
        return trend < -0.1  # Declining by 0.1 per period

    def check_slice(self, results: list[dict], threshold: float = 0.8) -> dict:
        """Find categories performing below threshold."""
        by_category = defaultdict(list)
        for r in results:
            by_category[r['category']].append(r['quality_score'])

        return {cat: np.mean(scores) for cat, scores in by_category.items()
                if np.mean(scores) < threshold * self.baseline_by_category[cat]}

Building Confidence in Changes

Before shipping a model or prompt change, build confidence through:

1. Offline evaluation: Run against golden set. All critical examples pass. Aggregate metrics stable or improved.

2. A/B test preview: Run both variants on a sample of production-like inputs. Compare quality distributions.

3. Staged rollout: 5% → 25% → 50% → 100%. Monitor metrics at each stage. Rollback capability ready.

4. Post-launch monitoring: Compare treated users against control. Watch for delayed effects.

The goal is to catch problems at the earliest, cheapest stage. A failure in offline evaluation costs minutes. A failure in production costs user trust.

---

Key Takeaways

1. Reference-based metrics are necessary but insufficient - ROUGE and BLEU work for sanity checks but correlate poorly with human judgment. Use them for screening, not optimization targets.

2. LLM-as-judge scales evaluation beyond human capacity - The key is calibration: validate that your judge correlates with human ratings before trusting it. Watch for biases around verbosity, position, and confidence.

3. Statistical rigor matters for A/B tests - LLM systems have higher variance than typical A/B scenarios. Calculate sample sizes properly, account for multiple variance sources, and don’t peek at results prematurely.

4. Production observability catches what testing misses - Log comprehensively, monitor quality on sampled traffic, detect drift before it becomes a crisis. User behavior signals (regeneration rate, abandonment) are valuable proxies.

5. Build golden sets with stratified difficulty - Easy examples that humans find trivial aren’t useful benchmarks. Include hard cases, edge cases, and cases targeting known failure modes.

---

Summary

LLM evaluation is fundamentally harder than traditional software testing. Outputs are non-deterministic, quality is subjective, and the space of possible responses is infinite. But with the right foundations, you can build evaluation systems that reliably measure what matters and catch problems before users do.

Key takeaways:

Reference-based metrics have their place but aren’t enough. ROUGE and BLEU work for sanity checks but correlate poorly with human judgment on quality. Use them for screening, not optimization.

LLM-as-judge scales evaluation beyond what’s practical with human raters. The key is calibration: validate that your judge correlates with human ratings before trusting it. Watch for biases around verbosity, position, and confidence.

Statistical rigor matters for A/B tests. LLM systems have higher variance than typical A/B scenarios. Calculate sample sizes properly, account for multiple sources of variance, and don’t peek at results prematurely.

Production observability catches what pre-launch testing misses. Log comprehensively, monitor quality on sampled traffic, detect drift before it becomes a crisis.

Evaluation is a continuous process, not a one-time gate. Build infrastructure for automated evaluation, trend monitoring, and rapid iteration. The teams that iterate fastest on evaluation often build the best products.

---

Practical Exercises

1. Calibrate an LLM judge: Collect 50 examples with human ratings spanning 1-5. Implement an LLM judge and compute calibration metrics. What’s your correlation? Where do judge and human disagree?

2. Build a golden set: For an application you’re working on (or a hypothetical one), curate 100 examples. Document the category coverage, difficulty distribution, and rationale for each critical example.

3. Run a sample size calculation: Given your expected effect size and metric variance, calculate required samples for 80% power. Double-check by running a simulation that measures actual achieved power.

4. Implement drift detection: Using real or synthetic interaction logs, implement input drift detection using embedding centroids. What threshold produces alerts that match actual distribution shifts?

5. Design an evaluation pipeline: Sketch the architecture for a complete evaluation system covering CI/CD, production monitoring, and human calibration. What components do you build first?

---

Self-Assessment Checkpoint

Conceptual Questions

Q1. [IC2] Why does BLEU score correlate poorly with human judgments of LLM output quality? Give a specific example where BLEU would mislead.

Answer

BLEU measures n-gram overlap with reference text. Problems: (1) Synonym penalty—“automobile” vs “car” scores 0 despite same meaning. (2) Fluent garbage—grammatically correct but meaningless text can score well. (3) Single-reference limitation—many valid answers exist, but BLEU punishes deviation from the one reference. Example: Reference: “The capital of France is Paris.” Model: “Paris is the capital city of France.” This semantically identical response gets penalized for word order and word choice (“city” is extra). LLM outputs often vary in valid ways that BLEU punishes.

Q2. [IC2] An LLM judge shows 85% agreement with human raters. Is this good enough? What additional information do you need?

Answer

85% agreement alone is insufficient. You need: (1) Human-human agreement baseline—if humans only agree 80%, 85% is excellent. If humans agree 95%, 85% is problematic. (2) Breakdown by category—85% overall might mask 95% on easy cases and 60% on hard ones. (3) Direction of disagreement—does the judge consistently rate higher/lower? (4) Failure mode analysis—are disagreements random or systematic (e.g., judge always prefers longer responses)? (5) Distribution of scores—if 90% of examples are clear-cut 1s and 5s, 85% on the ambiguous middle cases might actually be poor.

Q3. [Senior] How do you handle statistical significance when running A/B tests on LLM-powered features where variance is high?

Answer

Key approaches: (1) Increase sample size—use power analysis with realistic variance estimates from pilot data. LLM A/B tests often need 2-5x more samples than typical A/B tests. (2) Reduce variance through stratification—segment by user type, query difficulty, etc., and compute effects within strata. (3) Pre-registration—define success criteria and analysis plan before running to prevent p-hacking. (4) Multiple metrics—don’t rely on a single metric; look for consistent patterns. (5) Sequential testing with proper correction if you must peek at results. (6) Consider Bayesian approaches that provide credible intervals rather than just p-values.

Q4. [Senior] Describe how you would detect quality degradation in a production LLM system that doesn’t have clear ground-truth labels.

Answer

Without ground truth, use proxy signals: (1) User behavior—regeneration rate (users clicking “try again”), session abandonment, time-to-accept. Rising regeneration is a quality signal. (2) Automated quality scoring—run LLM-as-judge on sampled outputs, track score distributions. (3) Input drift detection—embedding centroids of input queries shifting indicates distribution changes that may surface quality issues. (4) Consistency checks—for same/similar inputs, are outputs becoming more variable? (5) Output pattern monitoring—response length distribution, refusal rate, citation count. (6) Downstream metrics—if the LLM feeds another system, monitor that system’s performance.

Q5. [Staff] You’re responsible for evaluation infrastructure serving multiple product teams. How do you design a platform that scales to 100 different LLM-powered features?

Answer

Design principles: (1) Centralized logging infrastructure—standardized schema for requests/responses across all features. (2) Pluggable judges—teams define domain-specific evaluation criteria, platform handles execution, calibration tracking, and reporting. (3) Golden set management—versioned, tagged test sets with tooling for curation and maintenance. (4) Evaluation-as-code—teams define evaluation in configuration (metrics, judges, thresholds), CI/CD runs automatically. (5) Dashboard standardization—common metrics views with drill-down capabilities. (6) Cross-team benchmarking—shared hard examples that all teams should handle. (7) Human labeling pool—shared annotation resources with quality control. (8) Cost tracking—evaluation budget allocation per team. Key: Provide enough standardization for consistency but enough flexibility for domain-specific needs.

Spot the Problem

Problem 1. [IC2] A team’s LLM judge prompt:

Rate the quality of this response from 1-5.

Response: {response}

Rating:

Why is this problematic?

Answer

Issues: (1) No rubric—what does 1 vs 5 mean? The judge will be inconsistent. (2) No context—quality depends on the task. A terse answer might be perfect for “What year?” but terrible for “Explain how…”. (3) No chain-of-thought—direct rating without reasoning leads to lower correlation with human judgment. (4) Missing the question/prompt—can’t evaluate relevance without knowing what was asked. Better: Include question, define rating criteria explicitly (1 = off-topic, 2 = partially relevant but wrong, etc.), ask for reasoning before the score.

Problem 2. [Senior] An A/B test setup:

# Running A/B test, checking results daily
while not statistically_significant(results):
    run_for_another_day()

if treatment_better(results):
    ship_treatment()

What’s wrong?

Answer

Classic peeking problem. Repeatedly checking for significance and stopping when you find it dramatically inflates false positive rate. If you check daily, you’re running multiple tests—p-value accumulates. The 5% false positive rate assumes a single test. Solutions: (1) Pre-specify sample size and only analyze at the end. (2) Use sequential testing methods (e.g., O’Brien-Fleming boundaries) that properly control error rate with interim analyses. (3) Use Bayesian approaches that are designed for continuous monitoring. Also: “treatment_better” needs to consider practical significance, not just statistical significance.

Problem 3. [Staff] A golden set for a customer support chatbot has 500 examples. All are questions that the team could answer correctly in 5 seconds. They report 98% accuracy. Why might this be misleading?

Answer

Selection bias—easy examples that humans find trivial aren’t challenging for the model either. The golden set doesn’t reflect the actual difficulty distribution of production queries. Missing: (1) Hard cases—complex questions, ambiguous situations, edge cases. (2) Adversarial cases—prompt injection attempts, out-of-scope questions, multi-turn context dependencies. (3) Realistic failure modes—questions about recent policy changes, product-specific details, cases requiring human escalation. 98% accuracy on easy cases might correspond to 60% on hard cases. A good golden set should stratify by difficulty and include cases that specifically target known failure modes.

Design Exercises

Exercise 1. [Senior] Design an evaluation system for a RAG-powered internal knowledge base. Users ask questions about company policies, and the system retrieves documents and generates answers. Define: the metrics you’d track, how you’d evaluate retrieval vs. generation separately, and how you’d monitor for quality issues in production.

Guidance

Metrics: (1) Retrieval—recall@k (are relevant docs retrieved?), precision@k (are retrieved docs relevant?). (2) Generation—faithfulness (does answer match retrieved docs?), relevance (does answer address the question?), completeness, citation accuracy. (3) End-to-end—answer correctness (using LLM judge or human review), user satisfaction (thumbs up/down, regeneration rate). Monitoring: (1) Sample and evaluate a percentage of production traffic daily. (2) Track query distributions—detect drift via embedding clustering. (3) Alert on low retrieval confidence scores. (4) Monitor edge cases—empty retrievals, very long responses, refusals. Evaluation pipeline: Separate retrieval eval (run queries, check if ground-truth docs are retrieved) from generation eval (provide fixed context, evaluate answer quality).

Exercise 2. [Staff] You’re introducing LLM-powered features to a regulated industry (healthcare/finance). The compliance team requires auditable evaluation. Design an evaluation system that: provides evidence of quality for regulators, tracks model changes over time, ensures evaluation isn’t gamed, and handles the reality that “correct” answers often require domain expert judgment.

Guidance

Requirements: (1) Audit trail—complete logs of what was evaluated, by whom, what the results were, versioned and immutable. (2) Evaluation governance—separation between those who build and those who evaluate. (3) Human calibration—domain experts validate LLM judge ratings, calibration tracked over time. (4) Golden set management—regulatory-relevant examples, versioned, with documentation of why each matters. (5) Change management—any prompt/model change triggers re-evaluation against golden set, with comparison to previous version. (6) Test coverage documentation—mapping of test examples to regulatory requirements. (7) Failure investigation process—documented procedure for analyzing failures, root cause, and remediation. Key: Treat evaluation as a regulated process itself, not just a technical tool.

---

Connections to Other Chapters

Evaluation is central to production AI systems. This chapter connects to:

• Chapter 7 (RAG Systems): RAG-specific evaluation metrics including retrieval recall, context relevance, and faithfulness.

• Chapter 8 (Agentic Systems): Evaluating agent behavior, tool use accuracy, and safety constraints.

• Chapter 20 (Responsible AI & Governance): How evaluation supports compliance, bias detection, and model governance.

• Chapter 30 (Data Architecture for AI): Managing evaluation datasets, versioning golden sets, and avoiding training-serving skew.

• Common Mistakes & Anti-Patterns (Appendix M): Evaluation anti-patterns and how to avoid them.

---

Further Reading

Essential

• Zheng et al. (2023), “Judging LLM-as-a-Judge” - Foundational work on using LLMs for evaluation, bias analysis, and calibration.
• Es et al. (2023), “RAGAS” - Evaluation framework specifically for RAG systems.
• Kohavi et al. (2020), “Trustworthy Online Controlled Experiments” - The definitive A/B testing guide from Microsoft.

Deep Dives

• Liu et al. (2023), “G-Eval” - Chain-of-thought prompting for evaluation with strong human correlation.
• Sculley et al. (2015), “Hidden Technical Debt in Machine Learning Systems” - Classic on operational ML challenges.
• Liang et al. (2022), “HELM” - Comprehensive multi-dimensional evaluation framework.