A build-it-yourself tour of large language models: rather than using pre-existing libraries, you code every piece in PyTorch — text tokenization and data sampling, the self-attention mechanism, a GPT model, the pretraining loop, and finally finetuning for classification and for following instructions. The result is a working, if small, GPT you understand end to end.
These pages are study notes — a chapter-by-chapter distillation of the key concepts, each paired with a quiz. They are a companion to the book, not a replacement for it. For the full code, figures, and exercises, read the original.
Original work & credit
Sebastian Raschka
ML researcher and educator; author of Machine Learning with PyTorch and Scikit-Learn. Notes here are based on the Manning MEAP edition.
- Title
- Build a Large Language Model (From Scratch)
- Publisher
- Manning Publications
- Published
- 2024
- ISBN
- 978-1-63343-716-6
These notes are independent, original summaries created for study purposes. All rights to the underlying work belong to Sebastian Raschka and Manning Publications. Please support the author by reading the full book.
Chapters
7 of manyUnderstanding Large Language Models
What an LLM is and why next-word prediction works, the pretraining→finetuning arc, the transformer (encoder/decoder, BERT vs. GPT), and the three-stage plan for building one.
Working with Text Data
Preparing text for training: word/subword tokenization, byte pair encoding, sliding-window data sampling, and turning tokens into embeddings (token + positional).
Coding Attention Mechanisms
Building self-attention from scratch: simplified attention, scaled-dot-product with trainable weights, causal (masked) attention with dropout, and multi-head attention.
Implementing a GPT Model from Scratch to Generate Text
Assembling the GPT: layer normalization, GELU and the feed-forward network, shortcut connections, the transformer block, and generating text from the untrained model.
Pretraining on Unlabeled Data
Training the GPT: cross-entropy loss and perplexity, the training loop, decoding strategies (temperature, top-k), saving/loading weights, and loading OpenAI's pretrained weights.
Finetuning for Classification
Adapting the pretrained GPT into a spam classifier: dataset prep, adding a classification head, finetuning selected layers, and evaluating classification accuracy.
Finetuning to Follow Instructions
Instruction finetuning: formatting an instruction dataset, batching with masking, training the model to follow instructions, and evaluating the responses.