A framework for building applications on top of foundation models (both LLMs and large multimodal models). It follows the real arc of an AI product: decide whether to build, understand how the model works, evaluate rigorously, then improve results through prompt engineering, context construction (RAG and agents), finetuning, and dataset engineering — before optimizing inference and closing the loop with user feedback.
These pages are study notes — a chapter-by-chapter distillation of the key concepts, each paired with a quiz. They are a companion to the book, not a replacement for it. For the full treatment, case studies, and references, read the original.
Original work & credit
Chip Huyen
Author of Designing Machine Learning Systems (O'Reilly); has built ML tooling and systems across industry for over a decade.
- Title
- AI Engineering
- Publisher
- O'Reilly Media, Inc.
- Published
- December 2024
- ISBN
- 978-1-098-16630-4
These notes are independent, original summaries created for study purposes. All rights to the underlying work belong to Chip Huyen and O'Reilly Media, Inc.. Please support the author by reading the full book.
Chapters
10 of manyIntroduction to Building AI Applications with Foundation Models
How language models grew into foundation models via self-supervision, the explosion of use cases, planning an AI product (use-case evaluation, the role of AI vs. humans, defensibility, expectations), and the three-layer AI engineering stack.
Understanding Foundation Models
The design decisions that shape a model: training-data recipes, model architecture and scale, post-training/alignment, and how sampling turns probabilities into responses — explaining hallucination and inconsistency.
Evaluation Methodology
How to evaluate open-ended responses: language-modeling metrics (perplexity), exact evaluation, AI-as-a-judge and its biases, and comparative evaluation (Elo/Bradley–Terry ranking).
Evaluate AI Systems
Turning methodology into a pipeline: choosing criteria, model selection, building an evaluation pipeline, and using data to compare and pick models for your application.
Prompt Engineering
What a prompt is and why prompt engineering works, best practices for writing prompts, context and in-context learning, and defending against prompt attacks.
RAG and Agents
Two context-construction patterns: retrieval-augmented generation (retrievers, chunking, embeddings) and agents (planning, tool use, reflection) — the well-proven and the frontier.
Finetuning
When to finetune and when not to, memory-efficient approaches (PEFT/LoRA), model merging, and how to reason about a model's memory footprint.
Dataset Engineering
Getting data for finetuning and beyond: data acquisition, annotation guidelines, data synthesis, and processing — plus what data quality means and how to evaluate it.
Inference Optimization
Making inference cheaper and faster at both the model level (quantization, distillation) and the inference-service level (batching, KV caching, parallelism).
AI Engineering Architecture and User Feedback
Assembling the concepts into an end-to-end application architecture, then designing a user-feedback system that collects useful signal while preserving a good experience.