Finetuning

Key concept · Sample efficiency and unlocking capabilities

Transfer learning improves sample efficiency: a good base model learns a task from far fewer examples (training from scratch for legal Q&A might need millions of examples; finetuning a base model, a few hundred). Ideally most of what's needed is already in the base model, and finetuning just refines behavior — OpenAI's InstructGPT framed finetuning as unlocking capabilities the model already has but that are hard to reach through prompting alone. Finetuning is an extension of pre-training; earlier chapters already met two kinds — supervised finetuning (SFT, on (input, output) pairs) and preference finetuning (RLHF/DPO). Self-supervised finetuning on cheap task-related data is continued pre-training.

Key concept · Reasons not to finetune

Much of what finetuning buys, prompting and context can buy more cheaply. Two big cautions: finetuning for one task can degrade performance on others (bad if your app takes diverse prompts — consider separate models, then merge them), and it carries a high up-front and ongoing cost — you need training data, ML training knowledge (learning rate, overfitting, evaluation), serving infrastructure, and a plan to keep up as new base models arrive faster than you can improve a finetuned one. Also beware the "you must finetune for domain-specific tasks" argument: as general models get stronger, they often beat domain-specific models (GPT-4 outscored BloombergGPT on financial benchmarks).

Key concept · Finetuning is for form, RAG is for facts

Once prompting is maxed out, choose your next step by your model's failure mode. Information-based failures (outputs factually wrong or outdated, or the model lacks private data) → RAG, which supplies knowledge and reduces hallucination. Behavior-based failures (right facts but wrong form — irrelevant, malformatted, not following a syntax like JSON or a domain-specific language) → finetuning, which teaches form and style. Studies found RAG often beats finetuning for up-to-date-knowledge tasks — and the two aren't exclusive: RAG on top of a finetuned model can help further.

Key concept · The adaptation workflow

There's no universal recipe, but a sensible order (with an evaluation pipeline in place from the start): (1) get the model working with prompting and version your prompts; (2) add examples (1–50); (3) if it fails on missing information, add RAG, starting with simple term-based retrieval; (4) then, by failure mode, either add more advanced RAG (information failures) or finetune (behavior failures); (5) combine RAG and finetuning for a further boost. Finetuning sits late because it's the most expensive step to build and maintain.

Key concept · Backpropagation, trainable vs. frozen parameters

Finetuning needs far more memory than inference because of how models train. Each step has a forward pass (compute output) and a backward pass (backpropagation — update weights from the error). A trainable parameter can be updated; a frozen parameter is kept fixed. During inference no parameters update (light); during finetuning, each trainable parameter also needs memory for its gradients and optimizer states (e.g. Adam keeps two extra values per parameter) plus activations. So the footprint is driven by three things: the number of parameters, the number of trainable parameters, and the numerical representation — and fewer trainable parameters → far less memory, which is the whole motivation for PEFT.

Key concept · Numerical precision and quantization

Each value's bit width sets its memory. FP32 = 4 bytes/parameter, so a 13B model's weights alone take ~52 GB; halve the width and it's ~26 GB. Quantization converts a model from more bits to fewer — a simple, effective way to shrink the footprint. Two rules: inference tolerates very low precision (16, 8, even 4 bits), while training is more sensitive, so it's usually done in mixed precision (some ops in 32-bit, some in 16- or 8-bit). Quantization is what lets a big base model be finetuned on modest hardware (e.g. QLoRA).

Key concept · LoRA — low-rank adaptation

The most popular PEFT method is LoRA. It freezes the original weights and learns a small update to them, factored into two low-rank matrices (a big weight update ΔW is approximated as A×B, where A and B are far smaller). Only A and B are trained — often well under 1% of the model's parameters — so gradients/optimizer states shrink dramatically. LoRA is prized for being memory-efficient, modular (train many small adapters for different tasks and swap them over one shared base model), and adding no inference latency once the learned update is merged back into the weights. QLoRA combines a quantized (e.g. 4-bit) frozen base with LoRA adapters, letting large models be finetuned on a single GPU.

Key concept · Adapters vs. soft prompts; PEFT vs. full finetuning

PEFT methods split into two families. Adapter-based (LoRA and kin) insert small trainable modules into the network. Soft-prompt-based (prefix tuning, prompt tuning, P-tuning) prepend trainable "virtual token" vectors to the input and train only those — no natural-language prompt, just learned embeddings. Versus full finetuning, PEFT uses a fraction of the memory and data, is much cheaper to store and serve (one base + tiny adapters), and usually reaches comparable quality — full finetuning still wins when you need to reshape the model deeply and have the resources.

Key concept · Ways to merge models

Three broad approaches. Summing / weight averaging: combine weights of models that share an architecture — simple linear or weighted averaging, SLERP (spherical interpolation), and task arithmetic (represent each finetuned skill as a "task vector" = finetuned − base, then add or even subtract skills; TIES and DARE reduce interference between vectors). Layer stacking: assemble a new model from layers of existing ones (used for mixture-of-experts "upcycling" and "frankenmerges"). Merging can produce a single model that's good at several tasks without paying to serve several models.

Key concept · Finetuning tactics

Running a finetuning project: pick a framework/API and a base model (start from the strongest that fits your budget and licensing; you can distill from a larger model into a smaller one). Tune the core hyperparameters — learning rate (the most important), batch size, and number of epochs — and watch for over/underfitting with your evaluation pipeline. A practical order some teams use is finetune-first vs. distillation-first depending on data and compute; whatever the path, benchmark every step against the same evaluation pipeline so you know each change actually helped.