Introduction to Building AI Applications with Foundation Models

Key concept · Tokens, tokenization, and vocabulary

The basic unit of a language model is the token — a character, a word, or (most often) a part of a word. Breaking text into tokens is tokenization, and the full set of tokens a model works with is its vocabulary. Tokens sit between characters and words: they let the model break words into meaningful components (cook + ing), keep the vocabulary smaller and the model more efficient, and handle unknown words gracefully. For GPT-4, a token averages roughly ¾ of a word, so ~100 tokens ≈ 75 words.

Key concept · Masked vs. autoregressive models

Two kinds of language model differ by what context they use to predict a token. A masked language model (e.g., BERT) fills in a blank using context from both before and after — good for non-generative tasks like classification and sentiment analysis. An autoregressive language model predicts the next token using only the preceding tokens, generating one token after another. Autoregressive models are the choice for text generation, so in this book "language model" means autoregressive unless stated otherwise.

Key concept · Self-supervision — why language models scaled

Traditional ML often relies on supervision: training on human-labeled examples, which is expensive and slow (labeling a million ImageNet images at 5¢ each ≈ $50,000). Language modeling is self-supervised: the training data provides its own labels, because each sequence supplies both the input context and the next token to predict. The sentence "I love street food." yields several training samples for free. Removing the labeling bottleneck is what let language models scale to the size behind the ChatGPT moment.

Key concept · Completion machines & generative AI

A language model is best understood as a completion machine: given a prompt, it predicts a plausible continuation. Completions are predictions based on probability — not guaranteed correct, which is exactly what makes these models both powerful and frustrating. Because a model can generate open-ended outputs from a finite vocabulary, it's generative — hence generative AI. Many tasks (translation, summarization, classification, coding) can be reframed as completion. A foundation model extends this idea: a large, general model — often multimodal — that serves as the shared base you adapt to many downstream applications.

Key concept · Coding leads; the enterprise prefers lower risk

Across surveys, coding is the single most popular use case (about 30% of the open-source applications studied), followed by conversational bots. But enterprises deploy differently from consumers: they move faster on internal-facing, lower-risk applications (internal knowledge management, text summarization) than on external-facing ones (customer chatbots), because internal apps build expertise while limiting the blast radius of data-privacy, compliance, and catastrophic-failure risks. Close-ended tasks like classification are also favored because they're easier to evaluate, making their risks easier to estimate.

Key concept · Use-case evaluation — three levels of motivation

Building an AI application is usually a response to risk or opportunity. Ordered high to low: (1) If you don't, competitors with AI can make you obsolete — existential, highest priority; (2) If you don't, you'll miss opportunities to boost profits and productivity (cheaper acquisition, better retention, faster operations); (3) You're unsure where AI fits, but don't want to be left behind — investing in understanding a transformational technology, if you can afford it. Once you have a reason, ask whether you must build it yourself or can buy.

Key concept · The role of AI and of humans

How central AI is shapes the requirements. Three axes: critical vs. complementary (Face ID fails without AI; Gmail's Smart Compose is optional) — the more critical, the higher the reliability bar; reactive vs. proactive (a chatbot reacts to a request; traffic alerts are proactive) — reactive features often need to be fast, proactive ones can be precomputed but need a higher quality bar since they're unsolicited; and dynamic vs. static (continually updated per-user vs. periodically updated for a group). Deciding how much humans stay in the loop is human-in-the-loop design — Microsoft's Crawl-Walk-Run framework increases automation gradually: crawl (human involvement mandatory), walk (AI interacts with internal staff), run (increased automation, including external users).

Key concept · Defensibility, expectations, and the last-mile challenge

A low entry barrier is a blessing and a curse: what's easy for you is easy for competitors, and if the underlying models get better, your thin layer can be subsumed. The three competitive moats are technology, data, and distribution — with foundation models, core tech converges and distribution favors incumbents, so a startup's edge is often the data flywheel from getting to market first. When you commit, set measurable business metrics and a usefulness threshold (quality, latency — TTFT/TPOT, cost per request, plus fairness/interpretability). Beware the last-mile challenge: initial demos are deceptively easy — "0 to 60 is easy; 60 to 100 is exceedingly hard" — the final gains against hallucinations and edge cases take the most time.

Key concept · What shifted — and what stayed the same

After Stable Diffusion and ChatGPT (2023), growth concentrated in the application and application-development layers; the infrastructure layer grew far less, because core needs — resource management, serving, monitoring — remain the same. So does much else: enterprise apps still solve business problems, so you still map business metrics to ML metrics, still evaluate rigorously, still care about latency and cost. The excitement is real, but many principles of productionizing AI carry straight over from traditional ML.

Key concept · AI engineering vs. ML engineering

Why "AI engineering" and not "ML engineering"? Working with foundation models differs enough to warrant the term. Traditional ML engineering leans on tabular data, feature engineering, and training models; AI engineering leans on prompt engineering, context construction, and parameter-efficient finetuning of models someone else trained. The center of gravity moves from building a model to adapting and evaluating one — though a real system often uses both foundation models and traditional ML, so both skill sets stay valuable.