Inference Optimization

Key concept · Compute-bound vs. memory-bandwidth-bound

Every workload has one of two bottlenecks. Compute-bound: time is set by the sheer number of calculations (e.g. image generation with Stable Diffusion). Memory-bandwidth-bound (often just "memory-bound"): time is set by how fast data moves between memory and processors. Which one you hit is captured by arithmetic intensity (operations per byte of memory access) and visualized on a roofline chart. The fix differs: compute-bound work wants more/faster compute (higher FLOP/s); memory-bound work wants higher bandwidth. Today, many transformer workloads are memory-bandwidth-bound.

Key concept · Prefill and decode — two different profiles

Autoregressive LLM inference has two steps with opposite characteristics. Prefill processes all input tokens in parallel — it's compute-bound, and its duration sets the time to the first token. Decode generates output tokens one at a time, each step loading the large weight matrices, so it's memory-bandwidth-bound. Because their profiles differ, production servers often decouple prefill and decode onto separate machines. The factors that drive an LLM server's bottlenecks are context length, output length, and batching strategy.

Key concept · Latency — TTFT, TPOT, and percentiles

For streaming generation, break latency into time to first token (TTFT) — how fast the first token appears, tied to the prefill step and input length — and time per output token (TPOT) (a.k.a. time between tokens / inter-token latency) — how fast each subsequent token comes. A useful identity: total latency ≈ TTFT + TPOT × (number of output tokens). Two apps with the same total can feel different — instant first token with slow generation, or a slower start with fast generation — and you can trade TTFT for TPOT by shifting compute between prefill and decode. Because latency is a distribution, don't report the average (outliers skew it): use percentiles — p50 (median), p90, p95, p99. A TPOT of ~120 ms (6–8 tokens/sec) beats human reading speed and is enough for most apps.

Key concept · Throughput, goodput, and the trade-off

Throughput is output tokens per second across all requests (tokens/s, or tokens/s/user; also requests/min). It's tied directly to cost — higher throughput means lower cost per token — so compare servers by cost per request rather than raw tokens/s (tokenizers differ). But maximizing throughput can wreck latency (batching more requests raises throughput while slowing each one). So teams track goodput: the requests per second that meet your latency SLO — if a server does 100 req/min but only 30 satisfy the "TTFT ≤ 200 ms, TPOT ≤ 100 ms" target, its goodput is 30. Online APIs optimize for latency (process on arrival, often with a streaming mode); batch APIs optimize for cost (relaxed latency allows heavy batching and cheaper hardware — often ~50% cheaper, in hours not seconds).

Key concept · Quantization, distillation, pruning

Three classic ways to shrink a model. Quantization — represent weights/activations in fewer bits (16 → 8 → 4) — is the most popular because it's simple, general, and needs no retraining; it makes the model smaller and faster and is the easiest big win. Distillation — train a small "student" to mimic a large "teacher," getting most of the quality at a fraction of the size. Pruning — remove parts of the network that contribute little (drop connections, or set weights to zero for sparsity). Quantization is usually the first lever to reach for.

Key concept · Attention and the KV cache

Much of an LLM's inference cost lives in attention. During decoding, the model would recompute the keys and values for every prior token at each step, so servers keep a KV cache — storing past keys/values to avoid recomputation. The catch: the KV cache grows with sequence length and batch size and often dominates memory. Optimizations target exactly this: FlashAttention restructures the attention computation to cut slow memory reads/writes (a big speedup), and PagedAttention manages the KV cache in fixed-size "pages" like an operating system's virtual memory (from vLLM), reducing waste and letting you batch more. Some models also cut cost with attention variants (e.g. multi-query / grouped-query attention) that share keys/values across heads.

Key concept · Batching

Running requests together amortizes the cost of loading weights, so batching is the single biggest throughput lever. Static batching waits to fill a fixed-size batch (simple, but early-finishing requests wait for the slowest). Dynamic batching forms a batch within a time window. Continuous batching (a.k.a. in-flight batching) is the LLM-specific winner: because generation is token-by-token, the server can add and evict requests every step as some finish, keeping the hardware busy without making short requests wait for long ones — a large throughput gain at little latency cost.

Key concept · Parallelism, prompt caching, speculative decoding

When a model or workload is too big for one device, split it: replica/data parallelism (copies handle different requests), tensor parallelism (split one layer's math across devices — lowers latency), and pipeline parallelism (put different layers on different devices). Two more high-value tricks: prompt caching reuses the computation for repeated prompt prefixes — long system prompts, a shared document, conversation history — so you don't re-prefill them every call (big savings for chat and RAG); and speculative decoding runs a small, fast "draft" model to propose several tokens that the large model then verifies in parallel, accelerating the memory-bound decode step with no loss in output quality. Underneath it all sit AI accelerators (GPUs/TPUs) chosen for their compute, memory, and bandwidth.