Understanding Large Language Models

Key concept · What "large" means, and the AI hierarchy

"Large" refers to two things at once: the model's size (tens or hundreds of billions of parameters — the adjustable weights tuned during training) and the immense dataset it's trained on. LLMs sit inside a nested hierarchy: artificial intelligence ⊃ machine learning (algorithms that learn from data instead of hand-written rules) ⊃ deep learning (neural networks with many layers, no manual feature engineering) — and LLMs are a specific application of deep learning. Because they generate text, LLMs are also a form of generative AI (GenAI).

Key concept · Next-word prediction and self-supervision

An LLM is trained on a deceptively simple task: predict the next word in a sequence. This is a form of self-supervised learning — the data provides its own labels, because the next word in the text is the label the model should predict. No human labeling is needed, so the model can learn from massive unlabeled corpora. This simple objective forces the model to learn grammar, context, structure, and even world knowledge — which is why researchers were surprised it produces such capable models.

Key concept · Pretraining creates a foundation model

Pretraining is the first stage: train the model on a large, diverse corpus of raw (unlabeled) text via next-word prediction. The "pre" signals that this is the initial phase; the result is a base model or foundation model (GPT-3 is the classic example) that can already do text completion and shows few-shot capabilities — learning a new task from just a handful of examples in the prompt. Pretraining is expensive (GPT-3's was estimated at ~$4.6M in compute), so the book trains a small model for learning and also shows how to load openly available pretrained weights.

Key concept · Finetuning: instruction vs. classification

Finetuning is the second stage: further train the pretrained model on a smaller, labeled dataset for a specific purpose. The two most common kinds: instruction-finetuning, where the labeled data is (instruction, answer) pairs (e.g. a translation request and its correct translation) — this is how ChatGPT was made from GPT-3; and classification finetuning, where the data is texts with class labels (e.g. emails labeled spam / not-spam). Finetuning needs far less data and compute than pretraining because it builds on capabilities the base model already has.

Key concept · Encoder, decoder, and self-attention

The original transformer has two submodules. The encoder processes the input text and turns it into numerical embedding vectors capturing its meaning; the decoder uses those to generate the output text one word at a time. The key component tying it together is self-attention, which lets the model weigh the importance of different words in a sequence relative to each other — capturing long-range dependencies and context. (Self-attention is coded from scratch in Chapter 3; data preprocessing in Chapter 2.)

Key concept · BERT vs. GPT

Two families adapted the transformer differently. BERT (Bidirectional Encoder Representations from Transformers) builds on the encoder and is trained on masked word prediction — filling in hidden words using context from both sides — making it strong at classification tasks (Twitter used it to detect toxic content). GPT (Generative Pretrained Transformer) builds on the decoder and is designed for generating text. A note on terminology: transformers and LLMs aren't synonyms — not all transformers are LLMs (some do vision), and not all LLMs are transformers (some use recurrent/convolutional architectures) — but this book uses "LLM" to mean a GPT-like transformer.

Key concept · Autoregressive generation and emergent behavior

GPT is an autoregressive model: it generates one word at a time, feeding each output back as input for the next prediction, so every new word is conditioned on all the words before it. Despite training only on simple next-word prediction, GPT models show emergent behavior — abilities never explicitly taught, like translation, classification, and summarization — that arise from exposure to vast, diverse data. GPT models are also large: the original transformer repeated its blocks 6 times, while GPT-3 has 96 transformer layers and 175 billion parameters.

Key concept · The three-stage plan for the book

The rest of the book codes a GPT from scratch in three stages. Stage 1 — Building an LLM: data preparation & sampling, the attention mechanism, and the LLM architecture (Chapters 2–4). Stage 2 — Pretraining: the training loop and model evaluation, producing a foundation model, plus loading openly available pretrained weights (Chapter 5). Stage 3 — Finetuning: take the pretrained model and finetune it into a classifier (Chapter 6) or a personal assistant / instruction-follower (Chapter 7). Everything runs on consumer hardware for educational purposes.