Finetuning to Follow Instructions

Key concept · The instruction dataset

The book uses a purpose-built dataset of 1,100 instruction–response pairs in JSON. Each entry has three fields: an instruction ("Identify the correct spelling…"), an optional input (extra context, sometimes empty), and the output (the desired response). The data is split 85/10/5 into train / test / validation. Unlike classification, the "label" here is a full free-form text response, not a class — which shapes everything about how the data is prepared.

Key concept · The Alpaca prompt style

Each entry is formatted as: a fixed preamble ("Below is an instruction that describes a task. Write a response that appropriately completes the request."), then ### Instruction: with the instruction, an optional ### Input: section (omitted when the input field is empty), and finally ### Response: with the desired output. This structured format gives the model clear delimiters to learn from. (An alternative shown is the Phi-3 style, which uses <|user|>/<|assistant|> tokens instead — a reminder that prompt formats vary across models.)

Key concept · Per-batch padding and shifted targets

An InstructionDataset formats each entry with the prompt template and pre-tokenizes it. The custom collate function then: pads every example in a batch to the length of the longest example in that batch (using token 50256) — a smarter approach than Chapter 6's dataset-wide padding, since it minimizes unnecessary padding (different batches can have different lengths). It builds the target as the input shifted one position right, with an end-of-text token appended, so the model learns next-token prediction on the response — and learns to emit an end-of-text token to signal it's done.

Key concept · The −100 placeholder

In each target sequence, all padding tokens except the first end-of-text token are replaced with the value −100. Why −100? PyTorch's cross_entropy has a default ignore_index of −100, so any target position set to −100 is excluded from the loss — the padding doesn't influence training. The first end-of-text token is kept (not masked) so the model still learns when to stop generating. (The book notes some implementations also mask the instruction/prompt portion of the target so the loss is computed only on the response; this book masks just the padding.) An optional allowed_max_length can truncate examples that exceed the model's 1,024-token context.

Key concept · Finetuning and extracting responses

The base model is a larger GPT-2 (355M "medium") — a more capable starting point than the 124M model — loaded with OpenAI's pretrained weights. Instruction finetuning runs the familiar loop (epochs → batches → reset → cross-entropy loss → backward → AdamW update), and the training/validation loss falls as the model learns to follow instructions. To evaluate, the finetuned model generates responses to the held-out test instructions; because the output includes the formatted prompt, you extract just the response portion (the text after ### Response:) for scoring.

Key concept · Evaluating with an LLM judge

Scoring free-form responses by hand doesn't scale, so the book uses another LLM as an automated judge: it runs Llama 3 (8B) locally via Ollama and prompts it to score each generated response against the correct answer on a 0–100 scale. Averaging those scores gives a single quality number for the finetuned model — the same "AI-as-a-judge" idea used across the industry. The finetuned model produces coherent, on-instruction answers, completing the journey: you've now built, pretrained, and finetuned a GPT entirely from scratch. Natural next steps the book points to include preference finetuning (e.g. DPO) to further align the model's tone and helpfulness.