Understanding Foundation Models

Key concept · Common Crawl and the English bias

Many models train on variations of Common Crawl (a nonprofit web crawl) or Google's cleaned subset C4. Its quality is questionable — clickbait, misinformation, fake news — yet it's used because it's available. English dominates it (~46% of the data, ~8× the next language, Russian), so general-purpose models work far better in English: on the MMLU benchmark, GPT-4 scores much higher in English than in under-represented languages like Telugu. Non-English use also costs more — tokenization is less efficient, so the same meaning takes far more tokens (median length 7 in English vs. 72 in Burmese), meaning higher latency and API cost.

Key concept · Domain-specific models

General models (Gemini, GPTs, Llamas) do well across common domains present in their data, but struggle on domains they never saw — drug discovery (protein/DNA/RNA data) or cancer screening (X-ray, fMRI) need data that's expensive, private, and absent from the public internet. That's why domain-specific models exist: DeepMind's AlphaFold (protein structures), NVIDIA's BioNeMo (drug discovery), Google's Med-PaLM 2 (medical). To build one you curate a very specific dataset.

Key concept · Attention — and the alternatives

Attention lets the decoder look at (attend to) all input tokens when generating each output token, weighting the relevant ones, and it enables parallel processing. The transformer has been so heavily optimized since 2017 that it's "sticky" — hard to displace. But alternatives are emerging for long-sequence efficiency: RWKV (parallelizable RNN), state space models (SSMs) — S4, H3, Mamba (linear-time scaling vs. the transformer's quadratic) — and hybrids like Jamba (interleaving transformer and Mamba layers).

Key concept · Three numbers that signal scale

A model's scale is captured by three numbers: number of parameters (a proxy for learning capacity); number of training tokens (a proxy for how much it learned — today's LLMs train on trillions); and FLOPs, floating point operations (a proxy for training cost). Watch two subtleties: a sparse model like a mixture-of-experts (MoE) activates only a subset of parameters per token (Mixtral 8×7B has 46.7B total but only ~12.9B active, so its cost matches a ~13B model); and FLOPs (compute for a task) differ from FLOP/s (a machine's peak speed).

Key concept · The Chinchilla scaling law

Given a fixed compute budget, what model and dataset size are best? The Chinchilla scaling law (DeepMind, 2022) found that for compute-optimal training, the number of training tokens should be roughly 20× the number of parameters (a 3B model wants ~60B tokens), and model size and data should scale together. Caveats: it optimizes for training, not inference (Llama chose smaller-than-optimal models for cheaper, more usable inference); and two scaling bottlenecks loom — running out of internet training data, and electricity. Related ideas: emergent abilities (capabilities that appear only at scale), inverse scaling (rare cases where bigger is worse), and scaling extrapolation (predicting good hyperparameters for a large model from small ones).

Key concept · Supervised finetuning (SFT) — demonstration data

SFT teaches the model to respond instead of merely continuing text. You show it demonstration data: (prompt, response) examples of the behavior you want — also called behavior cloning. The data should span the request types you care about (question answering, summarization, translation…). Unlike simple labeling, writing good demonstrations needs skill — for InstructGPT, generating one (prompt, response) pair could take up to 30 minutes and most labelers had college or advanced degrees. To cut this cost, teams increasingly turn to synthetic (AI-generated) data.

Key concept · Preference finetuning & RLHF

Demonstration teaches the model to converse, but not which conversations it should have. Preference finetuning aligns it to human preference. The classic algorithm is RLHF (reinforcement learning from human feedback), in two parts: (1) train a reward model that scores a response, and (2) optimize the model (often with PPO) to produce responses the reward model scores highly. The reward model is trained on comparison data — (prompt, winning_response, losing_response) — because it's easier and more reliable for labelers to compare two responses than to score one (pointwise). Newer approaches like DPO (used for Llama 3) skip the separate RL loop and are gaining traction; a simpler alternative some teams use is best-of-N — generate several outputs and keep the one the reward model likes best.

Key concept · Logits → softmax → probabilities

A neural network outputs a logit for each token in the vocabulary (the logit vector's size = vocabulary size). Logits aren't probabilities — they don't sum to one and can be negative — so a softmax layer converts them into a probability distribution. Greedy sampling always picks the highest-probability token; that's fine for classification but makes language generation repetitive and boring, so models instead sample from the distribution.

Key concept · Sampling strategies — temperature, top-k, top-p

Temperature rescales the logits before softmax: higher temperature (e.g. 0.7–1+) flattens the distribution for more creative/random output, lower (→0) sharpens it toward greedy and more deterministic output. Top-k restricts sampling to the k most-likely tokens. Top-p (nucleus) restricts it to the smallest set of tokens whose probabilities sum to p. All three trade off creativity against predictability and let you tune output to your application.

Key concept · Test-time compute and structured outputs

Because sampling is stochastic, generating multiple outputs and selecting among them — test-time compute, e.g. best-of-N with a scorer or a majority vote — raises quality at the cost of more inference. The sampling process can also be constrained to force structured outputs (valid JSON, a fixed set of labels) by restricting which tokens may be sampled — essential when a downstream system must parse the response.

Key concept · Why models hallucinate and are inconsistent

Sampling makes outputs probabilistic. Inconsistency — different answers to the same prompt, or the same answer phrased differently — follows directly from sampling from a distribution (lowering temperature reduces it). Hallucination — plausible but false statements — has two commonly cited causes: the model can't distinguish between data it was given and text it generated itself, and there's a mismatch between the model's internal knowledge and the labeler's knowledge (training it to assert things it doesn't "know" teaches it to make things up). Understanding this probabilistic nature is what lets you work with these behaviors rather than being surprised by them.