Evaluate AI Systems

Key concept · The four buckets of evaluation criteria

For a task like "summarize a legal contract," group criteria into four buckets: domain-specific capability (is the model good enough at law and language to do it at all?), generation capability (is the summary coherent and faithful?), instruction-following capability (is it in the requested format/length?), and cost and latency (how much, and how fast?). Every application weighs these differently; naming them upfront is what makes evaluation tractable.

Key concept · Domain-specific capability

A model can only do what its configuration and training data allow — no Latin in training, no Latin translation. Evaluate this with domain-specific benchmarks (public or private), usually via exact evaluation. Coding uses functional correctness, but you may also care about efficiency (BIRD-SQL scores runtime, not just correctness) and readability. Non-coding domains lean on close-ended tasks like multiple-choice — ~75% of lm-evaluation-harness tasks are MCQ — because they're easy to verify and score against a random baseline (four options → 25% baseline). Caveat: MCQ performance is sensitive to formatting, and testing the ability to pick a good answer isn't the same as testing the ability to generate one.

Key concept · Local vs. global factual consistency

Factual consistency splits by what you check against. Local: the output must be supported by a given context (a summary faithful to its document, a chatbot to company policy) — important for scoped tasks. Global: the output must agree with open knowledge — important for broad tasks like general chatbots and fact-checking. Verifying against explicit context is far easier than against open knowledge, where the hard part is determining what the facts even are.

Key concept · Ways to verify factual consistency

Given a context, the simplest verifier is an AI judge. More specialized options: self-verification (SelfCheckGPT — generate N responses and measure agreement; expensive), knowledge-augmented verification (SAFE decomposes a response into self-contained statements, sends fact-checking queries to a search engine, and rates each), and framing it as textual entailment — classify each (premise, hypothesis) pair as entailment (consistent), contradiction (inconsistent), or neutral (can't tell). Specialized entailment scorers (e.g. a fine-tuned DeBERTa) turn this into a classification task; TruthfulQA is a benchmark for it. For safety, general-purpose judges or smaller specialized toxicity classifiers (Perspective API, hate-speech detectors) flag toxicity, bias, and other harms.

Key concept · Measuring instruction-following

Two benchmarks frame it. IFEval focuses on automatically verifiable format instructions (25 types — keyword inclusion, length limits, bullet counts, JSON), scoring the fraction followed. INFOBench goes broader — content constraints, linguistic guidelines, style rules — and verifies each instruction with a set of yes/no decomposed questions (an AI or human answers them; score = criteria met ÷ total). A recurring gotcha: poor output can mean the model is bad or the instruction is bad, which is what makes this hard to measure. Since public benchmarks miss your specific instructions, curate your own (want YAML output, or no "As an AI model…"? test exactly that). Roleplaying — staying in character — is one common, hard-to-automate instruction type (RoleLLM, CharacterEval).

Key concept · Cost and latency (Pareto optimization)

A great-but-slow-and-costly model isn't useful; balancing quality, latency, and cost is Pareto optimization. Be explicit about what you can't compromise: set a hard latency ceiling, drop models that miss it, then pick the best of the rest. Latency has several metrics — time to first token (TTFT), time per token / between tokens, time per query — and since autoregressive models emit token-by-token, more output tokens means more latency (so prompt for conciseness). On cost, APIs charge per token (reduce token counts), while self-hosting cost is compute; per-token cost falls as you scale on your own hardware but stays flat on APIs, so the API-vs-self-host math changes with scale. A tip: distinguish must-haves from nice-to-haves — ask if users want lower latency and they always say yes, but high latency is usually an annoyance, not a dealbreaker.

Key concept · Build vs. buy — model APIs vs. self-hosting

The big fork is commercial API vs. self-hosting an open-weight model, traded off across several axes: data (APIs mean sending data out; self-hosting keeps it in but has fewer copyright/lineage checks), performance (the very best model is likely closed), functionality (APIs more often support scaling, function calling, structured outputs; self-hosting can expose logprobs and intermediate outputs, useful for classification and interpretability), cost (API fees vs. engineering effort), finetuning & control (self-hosting lets you finetune, quantize, and freeze a version; APIs bring rate limits, opaque model changes, and the risk of losing access), and on-device (only self-hosting runs locally/offline). The a16z finding: enterprises pick open models mainly for control and customizability, not just cost.

Key concept · Navigating public benchmarks and leaderboards

There are thousands of benchmarks (BIG-bench alone has 214), so leaderboards aggregate a subset — but aggregation raises two questions: which benchmarks to include, and how to combine them. Real leaderboards diverge (Hugging Face's Open LLM Leaderboard averaged six benchmarks; HELM used ten, only two overlapping) and often can't fully justify their picks. Two traps: benchmark correlation — strongly correlated benchmarks (WinoGrande, MMLU, ARC-C all test reasoning) are redundant and can exaggerate biases, so don't count them twice; and aggregation method — simple averaging treats an 80% on a hard benchmark the same as an easy one (HELM instead uses mean win rate). A high public rank doesn't guarantee fit, so build a custom leaderboard: gather benchmarks matching your criteria, prefer the latest (old ones saturate), and check each benchmark actually measures what it claims — and isn't contaminated (low perplexity ⇒ likely seen in training).

Key concept · Step 1 — evaluate all components (and all turns)

An app is usually a system of parts, so evaluate the end-to-end output and each component — a per-component score localizes failures the overall score hides. For anything conversational, distinguish turn-based evaluation (score each individual response) from task-based evaluation (did the whole multi-turn conversation accomplish the user's goal?) — a bot can give good-looking answers turn by turn yet fail the task.

Key concept · Step 2 — create an evaluation guideline

The most important — and most skipped — step. Spell out what the application should and shouldn't do, and write clear scoring rubrics with concrete examples of what a 1 vs. a 5 looks like, so scores are reproducible. Crucially, tie criteria to business metrics: don't just ask "is factual consistency high?" — decide what score the app must hit to actually be useful (its usefulness threshold), and what that means for the business.

Key concept · Step 3 — define methods, data, and check the pipeline itself

Pick a mix of methods per criterion (logprob-based, exact, AI judges), balancing cost and reliability, and curate evaluation data that mirrors production — enough examples for signal, and sliced (by user segment, input type, difficulty) so you can spot where the app fails instead of hiding it in an average. Then evaluate the evaluation pipeline itself: are its results reliable (reproducible, low-variance), do they correlate with the outcomes you care about, and is it worth its cost? An evaluation you can't trust is worse than none. Reliable evaluation, the author argues, is the bottleneck whose removal unlocks the next wave of AI applications.