Evaluation Methodology

Key concept · Four reasons evaluation got harder

(1) Intelligence outpaces easy judgment — anyone can grade a first-grader's math; almost no one can grade a PhD-level proof, and validating a summary may mean reading the whole book. (2) Open-endedness breaks ground truth — classification has a fixed set of correct answers to compare against; open-ended generation has too many valid responses to enumerate. (3) Models are black boxes — providers often hide architecture, data, and training, so you can only observe outputs. (4) Scope expanded — general-purpose models must be evaluated not just on known tasks but for discovering new capabilities and limitations.

Key concept · Benchmark saturation

Public benchmarks keep proving inadequate because models saturate them — hitting near-perfect scores fast. GLUE (2018) saturated in about a year, forcing SuperGLUE (2019); MMLU (2020) was largely superseded by MMLU-Pro (2024). Benchmarks must keep evolving to stay meaningful, which is itself a reason evaluation is a moving target.

Key concept · Entropy and cross entropy

Entropy measures how much information, on average, a token carries — equivalently, how hard the next token is to predict (a 2-token language has entropy 1 bit; a 4-token language, 2 bits). Cross entropy measures how hard it is for your model to predict a dataset, and it decomposes into two parts: the data's own entropy H(P), plus how far the model's learned distribution Q diverges from the true distribution P (the KL divergence). Train perfectly and cross entropy equals the data's entropy (divergence → 0). Cross entropy also tells you how well a model can compress text.

Key concept · BPC, BPB, and perplexity

Since models tokenize differently, per-token bits aren't comparable across models — so BPC (bits-per-character) and BPB (bits-per-byte) normalize to a common unit. Perplexity is the exponential of (cross) entropy — the effective number of options the model is choosing among (PPL = 2^entropy in bits, or e^entropy in nats). Lower is better. Rules of thumb: more structured data → lower perplexity, bigger vocabulary → higher perplexity, and longer context → lower perplexity.

Key concept · What perplexity is good for (and its limits)

Perplexity is a proxy for capability (bad next-token prediction usually means bad downstream results) and has two neat uses: data-contamination detection (abnormally low perplexity on a benchmark suggests it leaked into training data) and deduplication (add new data only if its perplexity is high, i.e. it's novel). Caveat: perplexity is a poor proxy after post-training — SFT and RLHF, and techniques like quantization, can raise perplexity even as task performance improves, because post-training optimizes for completing tasks, not predicting the next token.

Key concept · Functional correctness

Evaluate whether the output actually does the job. For code, run it against test cases — the standard metric is pass@k: generate k samples per problem, count the problem solved if any sample passes all its tests, and report the fraction solved (pass@1 ≤ pass@3 ≤ pass@10). The same idea covers any task with a measurable objective — a game bot judged by its score, a scheduler judged by energy saved. It's the most reliable signal when you can get it.

Key concept · Similarity to reference data

When you can't run the output, compare it to reference responses (ground truths / canonical responses) in (input, references) pairs — metrics that need references are reference-based; those that don't are reference-free. Four ways to measure similarity, in rising sophistication: (1) ask an evaluator (human/AI); (2) exact match — output equals a reference (good for short factual answers, brittle for anything open-ended); (3) lexical similarity — surface overlap via edit distance / n-grams (BLEU, ROUGE, METEOR, TER, CIDEr); (4) semantic similarity — closeness in meaning, by turning texts into embeddings and comparing them (e.g. cosine similarity, BERTScore).

Key concept · The reference-data trap

Similarity-to-reference is only as good as the references. It's bottlenecked by how much reference data you can generate; a correct answer can score low if no reference happens to resemble it (Adept's Fuyu wrote a correct caption but scored 0.4 because no reference mentioned "Big Ben"); and references can simply be wrong (the WMT 2023 task found many bad reference translations). Worse, higher lexical-similarity scores don't guarantee better answers — on HumanEval, BLEU scores for correct and incorrect code were similar, so optimizing for BLEU ≠ optimizing for functional correctness. Semantic similarity avoids some of this but depends entirely on the embedding model's quality.

Key concept · Why AI judges took off

They're fast, cheap, and flexible: you can ask a model to judge on any criteria — correctness, toxicity, repetitiveness, wholesomeness, hallucination — just by prompting, and they work without reference data, so they run in production where no ground truth exists. Studies show strong correlation with humans: on MT-Bench, GPT-4's agreement with humans (85%) exceeded human–human agreement (81%); AlpacaEval judges correlated ~0.98 with the human-voted Chat Arena. A bonus: an AI judge can explain its verdict, which helps you audit results.

Key concept · Judge subjectivity — the score isn't fixed

AI-as-a-judge is a subjective method: the score depends on which judge model you use and how you prompt it. That makes it flexible but also inconsistent — the same probabilistic nature that makes models useful means a judge can score the same response differently. The book's stance: even when an AI judge isn't as good as a human, it can still be good enough to guide development and give the confidence to ship — provided you understand its limitations, which the next chapter turns into a practical evaluation pipeline.

Key concept · The biases every AI judge carries

Know these so you can interpret and mitigate them. Self-bias: a judge favors its own outputs (GPT-4 favored itself by ~10%, Claude-v1 by ~25% in one study). Position bias: judges tend to favor the first answer in a pairwise comparison (mitigate by swapping orderings and averaging) — the mirror image of humans' recency bias. Verbosity bias: judges prefer longer answers even when a shorter one is more correct. Judges are also criteria-inconsistent across tools — "faithfulness" or "relevance" mean different things and use different scales in MLflow vs. Ragas vs. LlamaIndex, so scores aren't comparable. And a golden rule: don't trust a judge whose model and prompt you can't see, and keep the judge fixed so score changes reflect the app, not the judge.

Key concept · Win rate, rating algorithms, and ranking as prediction

The probability that model A beats model B is A's win rate over B. With many models you can't just count wins — you use a rating algorithm borrowed from sports and chess (Elo, Bradley–Terry, TrueSkill) to turn match outcomes into a score per model, then rank by score. (Chatbot Arena used Elo, then switched to Bradley–Terry because Elo was too sensitive to match order.) Crucially, ranking is a predictive problem: a ranking is "correct" if, for any pair, the higher-ranked model is more likely to win a future match — so a ranking's quality is judged by how well it predicts future outcomes, and there's no fixed ground truth. Don't confuse it with A/B testing, where a user sees one model at a time.

Key concept · Its challenges — and why it still has a future

Comparative evaluation is data-intensive (pairs grow quadratically; the transitivity assumption — A>B and B>C ⟹ A>C — cuts the work but may not hold for human preference), hard to standardize (crowdsourced votes can reward answers that merely sound better, and floods of trivial prompts pollute the ranking), and awkward for private models. It also gives relative, not absolute, signal: knowing B beats A 51% of the time doesn't tell you whether either is "good enough," so you still need absolute evaluation for a cost–benefit call. Yet it's hard to game (hence widely trusted), it never saturates as long as newer models arrive, and it captures the quality we actually care about — human preference — making it a strong complement to benchmarks (offline) and A/B testing (online).