Measuring how good the generated text is
To improve the model you first need a number that says how wrong it is. Generation is a three-step cycle — text → token IDs → model logits → back to text — and training aims to make the model assign high probability to the correct next token at each position.
Key concept · Cross-entropy loss = negative average log probability
For each input, the target is the input shifted by one (the next tokens). The model's softmax gives a probability for every vocabulary token at every position; you pick out the probabilities it assigned to the correct target tokens. The loss is built in six steps: logits → probabilities → target-token probabilities → take the log → average → negate. That negative average log probability is the cross-entropy loss. Training minimizes it — an untrained model has a high loss (~10.8 here); a perfectly trained one approaches 0. PyTorch's cross_entropy does all six steps in one call (working on flattened logits and targets).
Key concept · Perplexity
Perplexity is a more interpretable companion to the loss: perplexity = exp(loss). It's the effective vocabulary size the model is uncertain about at each step — a perplexity of 48,725 means the model is essentially unsure among ~48,725 tokens (nearly the whole 50,257-token vocabulary), i.e. it's guessing. Like the loss, lower is better; as training proceeds, both fall.
Splitting data and measuring both losses
The book trains on the same small public-domain story (The Verdict, ~5,145 tokens) so everything runs on a laptop in minutes — real pretraining is astronomically more expensive (training Llama 2 7B took ~184,320 A100-GPU-hours, roughly $690,000).
Key concept · Training vs. validation loss
The data is split — e.g. a train_ratio of 0.90 puts 90% in training and 10% in validation — and each part gets its own DataLoader (using the sliding-window loader from Chapter 2). Two helper functions compute the loss of a single batch and average the loss over a whole loader. You track both the training loss (on data the model learns from) and the validation loss (on held-out data) so you can watch for overfitting: if training loss keeps dropping while validation loss rises, the model is memorizing rather than generalizing.
How the weights actually get updated
Training an LLM uses the same loop as any deep network in PyTorch — just at larger scale.
Key concept · Epochs, batches, and the update step
The loop: for each epoch (one full pass over the training set), for each batch — (1) reset the gradients from the previous step; (2) compute the loss on the batch; (3) do a backward pass (backpropagation) to get the loss gradients; (4) update the model weights using those gradients via an optimizer (the book uses AdamW). Optional monitoring steps print the training/validation losses and generate a sample of text so you can watch the model improve. Over the epochs the loss drops and the model shifts from gibberish toward text that resembles the training data (with a small dataset it eventually overfits — memorizes — which is expected here).
Beyond greedy: temperature and top-k
Greedy decoding (always argmax) is deterministic and tends to be repetitive. Two knobs make generation more varied and natural, and they reduce verbatim memorization of the training data.
Key concept · Temperature scaling
Temperature rescales the logits before softmax. A temperature below 1 sharpens the distribution — the model becomes more confident and deterministic (approaching greedy as temperature → 0). A temperature above 1 flattens it — more diverse and creative, but riskier (it can pick implausible tokens). Instead of always taking the single most likely token, you now sample from the temperature-adjusted distribution, so the same prompt can yield different outputs.
Key concept · Top-k sampling
Top-k sampling guards against the downside of high temperature. Keep only the k highest-logit tokens, set all others to −∞ (so they get zero probability), then sample from the renormalized distribution of just those k. This preserves diversity while preventing the model from ever selecting a wildly implausible token. Top-k and temperature are commonly combined — top-k restricts the candidate set, temperature shapes how boldly you sample within it.
Persisting the model — and borrowing OpenAI's
Training is expensive, so you save what you've trained; and because the from-scratch architecture matches GPT-2, you can also load OpenAI's pretrained weights and skip pretraining entirely.
Key concept · Saving and resuming
Save the model's learned parameters with torch.save(model.state_dict()) and restore them with load_state_dict. To resume training later (not just run inference), also save the optimizer's state — AdamW keeps running statistics per parameter, and without them a resumed run would restart the optimizer's momentum from scratch.
Key concept · Loading OpenAI's pretrained GPT-2
OpenAI released the GPT-2 weights publicly (unlike GPT-3's). Because the book's GPTModel was built to the same architecture, you can download those weights and map them into your model — matching each weight tensor to the right layer — turning your from-scratch model into a genuinely capable one that now generates coherent text. This is the payoff: you understand every component and get a working model without paying for pretraining. These loaded weights are the starting point for the finetuning in Chapters 6 and 7.
Chapter 5 quiz
Fifteen questions on loss, perplexity, training, decoding, and weights. Answer first, then expand Show answer.
Part A · Multiple choice
The cross-entropy loss for an LLM is essentially:
- A The negative average log probability of the correct target tokens
- B The number of parameters
- C The vocabulary size
- D The batch size
Show answer
A. Minimized during training; PyTorch's cross_entropy computes it.
As a model trains well, its loss:
- A Approaches 0
- B Approaches infinity
- C Stays fixed
- D Becomes negative
Show answer
A. It assigns higher probability to the correct next tokens.
Perplexity is computed as:
- A exp(loss)
- B log(loss)
- C loss squared
- D 1 − loss
Show answer
A. The effective number of tokens the model is uncertain among.
Why track validation loss separately from training loss?
- A To detect overfitting (train loss falling while val loss rises)
- B To make training faster
- C It's identical to training loss
- D To increase the vocabulary
Show answer
A. Validation data is held out from training.
The four core steps inside each training batch are:
- A Reset gradients → compute loss → backward pass → update weights
- B Tokenize → embed → decode → print
- C Save → load → sample → repeat
- D Split → shuffle → batch → drop
Show answer
A. The book uses the AdamW optimizer for the update.
A temperature below 1 makes generation:
- A More deterministic (sharper distribution, closer to greedy)
- B More random
- C Slower
- D Longer
Show answer
A. Temperature above 1 flattens the distribution for more diversity.
Top-k sampling works by:
- A Keeping only the k highest-logit tokens and sampling from those
- B Always taking the top token
- C Averaging the top k
- D Removing the top k
Show answer
A. The rest are set to −∞ so they can't be chosen.
To resume training later (not just run inference), you should also save:
- A The optimizer state (e.g. AdamW's running statistics)
- B The tokenizer only
- C The dataset
- D Nothing extra
Show answer
A. Otherwise the optimizer's momentum restarts from scratch.
Part B · True or false
Lower perplexity indicates the model's predictions are closer to the actual distribution.
Show answer
True. Like the loss, lower is better.
Greedy (argmax) decoding tends to produce varied, non-repetitive text.
Show answer
False. It's deterministic and often repetitive; temperature/top-k add variety.
Because the from-scratch model matches GPT-2's architecture, OpenAI's pretrained weights can be loaded into it.
Show answer
True. This skips the expensive pretraining and yields coherent text.
On a tiny dataset, the model can overfit and memorize the training text.
Show answer
True. Expected here; decoding strategies help reduce verbatim memorization.
Part C · Short answer
What does a perplexity of ~48,000 tell you about an untrained GPT-2-vocabulary model?
Show answer
The model is essentially guessing — uncertain among ~48,000 tokens out of the 50,257-token vocabulary, i.e. it has learned almost nothing yet.
How do top-k and temperature complement each other?
Show answer
Top-k restricts sampling to the most plausible candidate tokens (preventing wild picks); temperature controls how boldly you sample within that set. Combined, they give diversity without implausibility.
Why is pretraining from scratch impractical for most people, and what's the book's workaround?
Show answer
Pretraining needs enormous compute (hundreds of thousands of GPU-hours, ~$690k for Llama 2 7B). The workaround: load OpenAI's publicly released pretrained GPT-2 weights into the matching from-scratch architecture.