A top-down view of a GPT model
A GPT is a deep network that generates text one token at a time. Its top-level flow: token embeddings + positional embeddings → dropout → a stack of transformer blocks → a final layer normalization → a linear output head that produces logits over the vocabulary. The book targets the smallest GPT-2, 124 million parameters.
Key concept · The GPT-124M configuration
The model is defined by a small config dictionary: vocab_size 50,257 (the BPE vocabulary), context_length 1,024 (max tokens, set by the positional embeddings), emb_dim 768 (embedding size), n_heads 12 (attention heads), n_layers 12 (transformer blocks stacked), drop_rate 0.1 (dropout), and qkv_bias False. A key idea: the model's real components are coded once and repeated — 12 identical transformer blocks — which is why a huge model is architecturally simple.
Key concept · Inputs, outputs, and logits
The model takes a batch of token-ID sequences and returns a tensor of logits with shape (batch, num_tokens, vocab_size) — for each position, an unnormalized score for every possible next token. For a batch of 2 texts × 4 tokens, the output is 2 × 4 × 50257. At the end of the chapter those 50,257-dimensional vectors are converted back into token IDs and decoded into words. "Parameters" are the trainable weights adjusted during training (e.g. a 2048×2048 layer holds ~4.19M of them).
Stabilizing training
Deep networks are hard to train because of vanishing and exploding gradients, which make weight updates unstable. Layer normalization is one of the main tools that keeps GPT training stable.
Key concept · Mean 0, variance 1 — plus scale and shift
Layer normalization adjusts a layer's activations to have a mean of 0 and a variance of 1 (unit variance), computed across the embedding/feature dimension for each token. This speeds convergence and makes training more reliable. It also adds two learnable parameters — a scale and a shift — so the network can undo or adjust the normalization if that helps. In GPT-2 and modern transformers, layer norm is applied before the multi-head attention and the feed-forward network (pre-LayerNorm), and once more before the final output layer.
The nonlinearity inside each block
Each transformer block contains a small fully-connected feed-forward network (FFN) that processes each token position independently, using the GELU activation rather than the more familiar ReLU.
Key concept · GELU and the expand-then-contract FFN
GELU (Gaussian Error Linear Unit) is a smooth activation function: unlike ReLU, which hard-zeros all negatives, GELU allows small negative values and has a smooth, curved shape — which tends to improve training for deep models. The feed-forward network is two linear layers with GELU between them, and it expands then contracts: the first layer projects the embedding up to 4× the embedding dimension (768 → 3072), GELU is applied, and the second layer projects back down (3072 → 768). This wider middle gives the model more capacity to learn richer representations while keeping the block's input and output the same size.
Residual connections and the transformer block
Stacking many layers reintroduces the vanishing-gradient problem. The fix, and the last building block, is the shortcut connection — and with it, the full transformer block can be assembled.
Key concept · Shortcut (residual) connections
A shortcut connection (a.k.a. skip or residual connection) adds a layer's input back to its output, creating a direct path for gradients to flow backward through the network. This prevents gradients from shrinking to nothing across many layers, so very deep models train effectively. The book demonstrates the effect by showing gradients that nearly vanish without shortcuts and stay healthy with them.
Key concept · The transformer block
The transformer block wires the pieces together in two sub-steps, each wrapped in a shortcut: (1) LayerNorm → multi-head causal attention (Chapter 3) → dropout, then add the block's input back; (2) LayerNorm → feed-forward network → dropout, then add again. Crucially, the block preserves the input shape — output dimensions equal input dimensions — which is exactly what lets you stack 12 of them without any glue code. This block is the repeated unit that makes up most of the GPT.
Assembling GPT and generating text
With every component in hand, the placeholder is replaced by the real GPTModel, and the chapter ends by generating text from it.
Key concept · The full GPTModel
The assembled GPTModel is: token embedding + positional embedding → dropout → n_layers transformer blocks (in sequence) → final LayerNorm → linear output head producing logits over the vocabulary. Counting parameters, the 124M-config model actually has ~163M trainable parameters unless weight tying is used (sharing the token-embedding matrix with the output head) — the trick GPT-2 uses to reach the advertised 124M. The book also computes the model's memory footprint from its parameter count.
Key concept · Generating text token by token
Generation is a loop. Feed the current token sequence to the model; take the logits for the last position only (the prediction for the next token); optionally apply softmax; pick the token with the highest probability (greedy / argmax decoding); append it to the sequence; and repeat for as many tokens as you want. The input is truncated to the model's context length if it grows too long. Because this model is untrained, its output is gibberish — but the generation machinery is complete; Chapter 5 trains the weights so the output becomes coherent.
Chapter 4 quiz
Fifteen questions on the GPT architecture, layer norm, GELU/FFN, shortcuts, and generation. Answer first, then expand Show answer.
Part A · Multiple choice
Why is a huge GPT model architecturally simple?
- A Its core components (transformer blocks) are coded once and repeated
- B It has only one layer
- C It uses no attention
- D It's written in one line
Show answer
A. GPT-124M stacks 12 identical transformer blocks.
In GPT-124M, emb_dim, n_heads, and n_layers are:
- A 768, 12, 12
- B 50257, 1024, 0.1
- C 256, 8, 4
- D 1600, 25, 48
Show answer
A. vocab_size 50257, context_length 1024, drop_rate 0.1.
The model's output logits have shape:
- A (batch, num_tokens, vocab_size)
- B (batch, 1)
- C (vocab_size,)
- D (num_tokens, emb_dim)
Show answer
A. A next-token score for every position over the whole vocabulary.
Layer normalization adjusts activations to have:
- A Mean 0 and variance 1 (unit variance)
- B Mean 1 and variance 0
- C All zeros
- D Random values
Show answer
A. Plus learnable scale and shift parameters.
Layer norm helps mainly by:
- A Stabilizing training against vanishing/exploding gradients
- B Making the model smaller
- C Generating text
- D Tokenizing input
Show answer
A. Speeding convergence and reliability.
GELU differs from ReLU in that it:
- A Is smooth and allows small negative values
- B Zeros all inputs
- C Is linear
- D Has no derivative
Show answer
A. A smoother curve that tends to help deep-model training.
The feed-forward network in a transformer block:
- A Expands to 4× the embedding dim, applies GELU, then contracts back
- B Shrinks then grows
- C Keeps one fixed width
- D Has no activation
Show answer
A. 768 → 3072 → 768 in GPT-124M.
A shortcut (residual) connection:
- A Adds a layer's input back to its output, easing gradient flow
- B Deletes a layer
- C Doubles the parameters
- D Removes attention
Show answer
A. Prevents gradients from vanishing across many layers.
Greedy text generation picks the next token by:
- A Taking the argmax of the last position's logits
- B Choosing randomly
- C Averaging all tokens
- D Using the first token
Show answer
A. Append it and repeat.
Part B · True or false
A transformer block preserves its input's shape, so blocks can be stacked directly.
Show answer
True. Output dimensions equal input dimensions.
Each transformer block wraps both the attention and the FFN sub-step in a shortcut connection.
Show answer
True. LayerNorm → attention → dropout → add; LayerNorm → FFN → dropout → add.
Weight tying lets the 124M config reach its advertised parameter count.
Show answer
True. Sharing the token-embedding matrix with the output head; without it the model has ~163M parameters.
The untrained GPT model produces coherent text out of the box.
Show answer
False. Output is gibberish until the model is trained in Chapter 5.
Part C · Short answer
Describe the data flow through the full GPTModel.
Show answer
Token embeddings + positional embeddings → dropout → n_layers transformer blocks → final layer normalization → linear output head → logits over the vocabulary.
List the two sub-steps inside a transformer block.
Show answer
(1) LayerNorm → multi-head causal attention → dropout → add shortcut; (2) LayerNorm → feed-forward (GELU) network → dropout → add shortcut.