Dataset Engineering

Key concept · Data format follows the training phase

What data you need depends on what you're teaching. Self-supervised finetuning (continued pre-training) wants sequences of text. Supervised finetuning wants (instruction, response) pairs. Preference finetuning wants (instruction, winning response, losing response). A reward model uses that comparison format or ((instruction, response), score). Training data must exhibit the behaviors you want the model to learn — and teaching hard behaviors is harder: chain-of-thought data needs step-by-step responses (tedious to write, so CoT datasets are rarer), and tool-use data needs the action sequences an agent should take (often captured by simulation because what's efficient for a human — a browser — differs from what's efficient for a model — an API).

Key concept · Single-turn vs. multi-turn, and unlearning

For conversational apps, decide whether you need single-turn data (train the model to answer individual instructions — simpler, easier to get) or multi-turn data (teach it to accomplish a task over back-and-forth — clarify intent, take corrections; harder, often needs purpose-built scenarios). And curation isn't only about adding data to teach new behaviors — it's also about removing data to make a model unlearn bad ones.

Key concept · What makes data high-quality

High-quality data tends to be: relevant (to the task you're training for), aligned with your task requirements (e.g. factually correct for a Q&A app, creative for a story app), consistent (annotators would agree on the label; contradictory labels confuse the model), correctly formatted (matching the model's expected structure), sufficiently unique (deduplicated — duplicates waste compute and can bias the model), and compliant (with privacy, copyright, and your policies). Poor data does real damage — it can increase biases and hallucinations — so bad data is worse than less data.

Key concept · Coverage and quantity

Coverage is diversity: the data should span the range of inputs, patterns, and edge cases your model will face in production — a model only learns behaviors present in its data. Quantity — how much you need — has no single answer; it depends on the finetuning method (full finetuning needs far more than PEFT/LoRA), the task complexity, and how good the base model already is. Practical guidance: start with a small high-quality set (finetuning experiments can begin with as few as ~50–100 examples), measure, and add data while it still helps — returns diminish, and beyond a point more data barely moves the needle while quality problems keep hurting.

Key concept · Why annotation is hard

Good annotation demands a clear annotation guideline — the same document that anchors evaluation (Chapter 4) — defining what a correct response looks like, with examples, so different annotators agree. It gets harder for complex behaviors: writing a single high-quality (instruction, response) pair for InstructGPT could take up to 30 minutes and needed educated labelers; CoT and tool-use examples are harder still. And human demonstrations aren't always ideal for a model — a human might use a browser where a model should call an API — so teams increasingly turn to simulation and synthetic data for behaviors humans can't demonstrate efficiently.

Key concept · Why synthesize, and how

Reasons to generate data: boost quantity cheaply, improve coverage of rare or dangerous edge cases, protect privacy (synthetic stand-ins for sensitive records), and distill a stronger model's behavior into a smaller one. Techniques span a spectrum: rule-based generation (templates, procedural generation, and augmentation — paraphrasing, back-translation, perturbations that create variants of existing data), simulation (generate tool-use/agent trajectories in a simulated environment), and AI-powered generation (have a model write instructions and responses — e.g. self-instruct, or generate from a more capable "teacher" model).

Key concept · Distillation — and the limits of synthetic data

Model distillation trains a smaller "student" on a larger "teacher" model's outputs, transferring capability at lower serving cost (though a provider's terms may forbid using its outputs to train competitors — check licensing). But synthetic data has real limits: quality is hard to control, so it needs verification (functional checks, AI or human judges, filtering) before use; it can propagate the generator's biases and errors; and training a model repeatedly on AI-generated data risks model collapse — quality degrades as the model learns from its own (or another model's) outputs rather than real-world signal. The rule: synthesize to augment real data, verify what you generate, and don't let synthetic data crowd out the real thing.

Key concept · Inspect, deduplicate, clean & filter, format

A practical pipeline: (1) Inspect the data first — understand its distribution, spot obvious errors and surprises before training on it (this always pays off). (2) Deduplicate — remove exact and near-duplicates, which waste compute, skew the distribution, and can leak test data into training. (3) Clean and filter — remove low-quality, malformatted, toxic, or non-compliant examples and PII; you can also filter for quality, keeping only the best examples (often more valuable than adding more). (4) Format — put the data in exactly the structure the model expects, matching its chat template (a mismatch causes the silent failures from Chapter 5). One more knob for instruction finetuning: the prompt loss weight — how much prompt tokens (vs. response tokens) contribute to the loss, commonly ~10%, so the model learns mostly from responses.