Measure before you tune
Prompt evaluation turns "does this prompt seem better?" into a number. Instead of eyeballing a few outputs, you test a prompt against many cases and score them, so changes are driven by evidence.
Key concept · A typical eval workflow
(1) Assemble a test dataset — representative inputs with, where possible, expected outputs; you can even have Claude generate test datasets to get coverage fast. (2) Run the eval — send each case through your prompt and collect outputs. (3) Grade the outputs against your criteria and compute a score. (4) Iterate — change one thing, re-run, compare. The dataset is the fixed yardstick that makes prompt changes comparable.
Two ways to score an output
How you grade depends on whether "correct" is exact or judgment-based.
| Method | How it works | Best for |
|---|---|---|
| Code-based grading | Deterministic checks in code — exact match, regex, JSON schema validity, value ranges. | Structured or verifiable outputs where right/wrong is precise. |
| Model-based grading | A second Claude call judges the output against a rubric (accuracy, tone, completeness). | Open-ended text where quality is a judgment call. |
Key concept · Match the grader to the task
Use code-based grading whenever the answer is checkable programmatically — it's fast, free, and unambiguous. Use model-based grading (an "LLM as judge") for the many tasks where correctness is subjective; give the grader a clear rubric and, ideally, examples of good and bad. Many real eval suites combine both: code checks the format, a model judges the substance.
Techniques that reliably help
With an eval in place, prompt engineering is designing effective instructions — and a handful of techniques do most of the work.
Key concept · Clear, specific, structured, exemplified
Be clear and direct — say exactly what you want, as you would to a new colleague; don't make Claude guess. Be specific — spell out format, length, audience, and constraints. Structure with XML tags — wrap inputs and sections (<document>…</document>, <instructions>…</instructions>) so Claude can tell context from task. Provide examples (few-shot) — showing two or three input→output pairs is often the single biggest quality lever. Layer these together, measure the effect on your eval, and keep what moves the score.
Chapter 20 quiz
Thirteen questions on evals, grading, and prompting. Answer first, then expand Show answer.
Part A · Multiple choice
Why build a prompt eval?
- A To measure prompt quality so changes are evidence-driven, not eyeballed
- B To slow development down
- C To pick a model
- D To get an API key
Show answer
A.
The fixed yardstick that makes prompt changes comparable is:
- A The test dataset
- B The API key
- C The temperature
- D The model name
Show answer
A.
Who can help generate a test dataset quickly?
- A Claude itself
- B Only manual authoring
- C The API key
- D No one
Show answer
A.
Code-based grading is best for:
- A Structured/verifiable outputs where right vs wrong is precise
- B Subjective tone judgments
- C Creative writing quality
- D Nothing
Show answer
A.
Model-based grading uses:
- A A second Claude call judging against a rubric
- B A regex
- C A schema validator
- D Exact string match
Show answer
A. "LLM as judge" for subjective quality.
Which prompting technique is often the single biggest quality lever?
- A Providing examples (few-shot)
- B Raising temperature
- C Removing the system prompt
- D Using a longer key
Show answer
A.
Why wrap inputs in XML tags?
- A So Claude can distinguish context from task/instructions
- B To make the prompt shorter
- C To change the model
- D No reason
Show answer
A. e.g. <document>…</document>.
Part B · True or false
Many eval suites combine code-based and model-based grading.
Show answer
True. Code checks format; a model judges substance.
"Be clear and direct" means saying exactly what you want, not making Claude guess.
Show answer
True.
The best way to compare two prompts is to eyeball a couple of outputs.
Show answer
False. Run both against the same eval dataset and compare scores.
You should change several things at once between eval runs to move faster.
Show answer
False. Change one thing, re-run, compare — so you know what caused the change.
Part C · Short answer
Name the two grading methods and when to use each.
Show answer
Code-based (verifiable/structured outputs) and model-based (subjective, judgment-based quality).
Name the four core prompt-engineering techniques.
Show answer
Be clear and direct, be specific, structure with XML tags, and provide examples.