Turning text into vectors
Neural networks can't process raw text — it's categorical, not numeric. So text must be converted into continuous-valued vectors, a process called embedding: a mapping from discrete objects (words, images, documents) to points in a continuous vector space that a network can operate on.
Key concept · Word2Vec and learned embeddings
An early, influential approach is Word2Vec, which trained a network to predict a word's context (or vice versa), on the principle that words appearing in similar contexts tend to have similar meanings — so similar terms cluster together in the embedding space. But LLMs don't use pretrained Word2Vec vectors; they learn their own embeddings as part of the input layer, updated during training. The advantage: embeddings are optimized for the specific task and data at hand. Embedding dimensionality is a trade-off — higher captures more nuance but costs compute. GPT-2 (small) uses 768 dimensions; the largest GPT-3 uses 12,288.
Splitting text and mapping it to integers
The first concrete step is tokenization: splitting text into tokens (words or punctuation). Then those tokens are mapped to integer token IDs — an intermediate step before embedding. (The book demonstrates on Edith Wharton's public-domain story The Verdict.)
Key concept · The vocabulary, encoding, and decoding
To turn tokens into IDs you first build a vocabulary: take all unique tokens, sort them, and assign each a unique integer. A tokenizer then offers two methods — encode (text → token IDs, via the vocabulary) and decode (token IDs → text, via the inverse vocabulary). A design choice along the way: whether to keep or drop whitespace (dropping saves memory; keeping matters for whitespace-sensitive text like Python code).
Key concept · Special context tokens
A simple word-level vocabulary breaks on any word it hasn't seen (a KeyError) — which motivates special tokens. <|unk|> stands in for unknown / out-of-vocabulary words; <|endoftext|> is inserted between unrelated documents so the model learns they're separate sources. Other schemes add [BOS] (beginning of sequence), [EOS] (end of sequence, like <|endoftext|>), and [PAD] (padding, to make batched texts equal length). GPT models keep it minimal — they use only <|endoftext|> (for both separation and padding, since padded tokens are masked) and no <|unk|> at all, because their tokenizer handles unknown words differently.
The tokenizer GPT actually uses
Instead of whole-word tokens, GPT-2, GPT-3, and the original ChatGPT use byte pair encoding (BPE), which breaks words into subword units. The book uses OpenAI's efficient tiktoken library rather than reimplementing it.
Key concept · How BPE handles any word
BPE builds its vocabulary bottom-up: start with all individual characters, then iteratively merge the most frequent adjacent pairs into subwords (e.g. "d"+"e" → "de", common in "define", "depend"), governed by a frequency cutoff. The payoff: an unknown word like "someunknownPlace" is simply split into known subwords or characters — so BPE needs no <|unk|> token and can encode any text. The GPT-2 BPE vocabulary has 50,257 tokens, with <|endoftext|> assigned the largest ID (50256).
Input–target pairs with a sliding window
Because LLMs are trained by next-word prediction, the training data is a set of input–target pairs where the target is the input shifted by one position. A sliding window moves across the tokenized text to generate these pairs.
Key concept · Context size, stride, and the shift-by-one target
For a chunk of tokens of length context size (a.k.a. max_length), the input is x = tokens[:n] and the target is y = tokens[1:n+1] — every target token is the "next word" the model should predict for the input up to that point. The stride is how far the window moves between chunks: stride 1 shifts by one token (lots of overlap); setting stride = context size gives non-overlapping chunks, which uses the data fully while avoiding overlap that could increase overfitting. Larger input sizes (256+) are typical; small ones are just for illustration.
Key concept · Dataset and DataLoader
The pairs are packaged with PyTorch's Dataset (GPTDatasetV1 tokenizes the text and stores input/target chunks) and DataLoader, which serves them in batches of tensors. Options include batch_size (a memory/noise trade-off — small batches use less memory but give noisier updates), shuffle, and drop_last (drop a final undersized batch to avoid loss spikes). Each batch yields an input tensor and a target tensor, e.g. shape 8 × 4 for 8 samples of 4 tokens.
The final input representation
The last step converts token IDs into the vectors the LLM's layers consume — and adds position information the model would otherwise lack.
Key concept · The embedding layer is a lookup table
A token embedding layer (torch.nn.Embedding(vocab_size, output_dim)) is a weight matrix with one row per vocabulary token and one column per embedding dimension. It's initialized with random values and optimized during training. Embedding a token ID is just a lookup: retrieve the matching row — equivalent to (but far more efficient than) one-hot encoding followed by a matrix multiply, and, being a neural-network layer, trainable via backpropagation. Embedding a batch of IDs yields a tensor like 8 × 4 × 256 (batch × tokens × embedding dim).
Key concept · Positional embeddings
A token embedding maps the same ID to the same vector regardless of position — but self-attention (Chapter 3) is also position-agnostic, so the model has no sense of word order. The fix is to add positional embeddings. Absolute positional embeddings assign a distinct vector to each position (1st, 2nd, …); relative ones encode "how far apart" tokens are, generalizing better to varying lengths. GPT uses learned absolute positional embeddings (optimized during training, not fixed sinusoids). The final input embeddings = token embeddings + positional embeddings (same dimension, added element-wise) — the input to the main LLM modules, built next in Chapter 3.
Chapter 2 quiz
Fifteen questions on embeddings, tokenization, BPE, data sampling, and positional embeddings. Answer first, then expand Show answer.
Part A · Multiple choice
Why must text be converted into embeddings before an LLM can use it?
- A Neural networks operate on numbers, not raw categorical text
- B Embeddings are shorter
- C Text is illegal input
- D To save disk space only
Show answer
A. An embedding maps discrete tokens to continuous vectors.
Unlike Word2Vec, an LLM's word embeddings are:
- A Learned as part of the model and optimized during training
- B Fixed and pretrained
- C Hand-written
- D Never updated
Show answer
A. Optimized for the specific task and data.
The two methods of a tokenizer are:
- A encode (text → token IDs) and decode (token IDs → text)
- B train and test
- C read and write
- D forward and backward
Show answer
A. Decode uses the inverse vocabulary.
The <|endoftext|> token is used to:
- A Separate unrelated documents (and, for GPT, act as padding)
- B Mark unknown words
- C Start every sentence
- D Encode positions
Show answer
A. GPT uses only this special token; unknown words are handled by BPE.
Byte pair encoding handles an unknown word by:
- A Splitting it into known subword units or characters
- B Replacing it with <|unk|>
- C Deleting it
- D Erroring out
Show answer
A. So BPE needs no <|unk|> token and can encode any text.
The GPT-2 BPE vocabulary size is:
- A 50,257
- B 1,130
- C 768
- D 12,288
Show answer
A. With <|endoftext|> as the largest ID (50256).
For next-word-prediction training, the target sequence is:
- A The input shifted by one position
- B The input reversed
- C A random sequence
- D Identical to the input
Show answer
A. y = tokens[1:n+1] for input x = tokens[:n].
Setting the sliding-window stride equal to the context size:
- A Produces non-overlapping chunks (uses data fully, less overfitting risk)
- B Skips half the data
- C Doubles the vocabulary
- D Has no effect
Show answer
A. Stride 1 instead shifts by one token, with heavy overlap.
A token embedding layer is essentially:
- A A lookup table (weight matrix) retrieving a row per token ID
- B A tokenizer
- C A loss function
- D A dataloader
Show answer
A. Equivalent to one-hot + matmul, but more efficient and trainable.
Part B · True or false
A token embedding layer starts with random weights that are then trained.
Show answer
True. The embeddings are optimized during LLM training.
Token embeddings alone give the model a sense of word order.
Show answer
False. The same ID maps to the same vector regardless of position — hence positional embeddings.
GPT uses learned absolute positional embeddings rather than fixed sinusoidal ones.
Show answer
True. They're optimized during training.
A smaller batch size uses less memory but gives noisier model updates.
Show answer
True. Batch size is a trade-off hyperparameter.
Part C · Short answer
How are the final input embeddings formed?
Show answer
Input embeddings = token embeddings + positional embeddings (same dimension, added element-wise).
List the pipeline from raw text to input embeddings.
Show answer
Raw text → tokenize → map tokens to token IDs (via vocabulary / BPE) → token embeddings (lookup) → add positional embeddings → input embeddings.