The first two steps of the architecture
The book grows the architecture from the simplest possible loop — query → model → response — adding one component at a time as the application demands. The first two additions are context and guardrails.
Key concept · Step 1 — enhance context
Give the model the information it needs via context construction — RAG (text/image/tabular retrieval), tools that gather information (web search, news, weather APIs), and query rewriting. This is feature engineering for foundation models, and it's so central that providers support it (file uploads, tool use) — though they differ in how many documents you can upload, chunking options, and whether tools run in parallel.
Key concept · Step 2 — put in guardrails
Guardrails protect you and your users wherever there's risk, in two places. Input guardrails guard against leaking private information to external APIs (an employee pastes a secret; a tool pulls private data into context) — detect and redact sensitive data, e.g. mask a phone number as [PHONE NUMBER] and restore it later via a reverse PII dictionary — and against bad prompts (jailbreaks, injection). Output guardrails catch failures and set a policy to handle them: quality failures (malformatted output, hallucination, generally bad answers) and security failures (toxicity, leaked private data, brand-risk responses). Mitigations include retry logic (models are probabilistic — try again; parallel calls keep latency down) and human fallback for tricky cases. Mind two trade-offs: a reliability-vs-latency tension (guardrails add latency), and that output guardrails fit awkwardly with streaming (you may show tokens before you can score them).
Steps 3 and 4 — managing many models and cutting latency
As an app grows to use several models, you add infrastructure to route between them and to reuse work.
Key concept · Step 3 — model router and gateway
A router sends different queries to different solutions — a specialized model for troubleshooting, a cheaper model for simple queries, a human for billing disputes. It's usually an intent classifier (predict what the user wants; also detect out-of-scope queries and ambiguous ones needing clarification) or a next-action predictor (should the agent search or run code next?). Routers are built from small, fast, cheap models. A gateway is a unified interface to many models (self-hosted and commercial): it centralizes access control and cost management (no sharing org API keys), fallback policies for rate limits/outages, plus load balancing, logging, and analytics — and when a model API changes you update only the gateway, not every app.
Key concept · Step 4 — reduce latency with caches
Beyond the inference caches of Chapter 9 (KV cache, prompt cache), system caching reuses whole results. Exact caching: reuse a stored result only for an identical request (a cached product summary, or a cached vector-search result). Semantic caching: reuse a result for a semantically similar query (via embeddings) — powerful but riskier, since it depends on judging similarity correctly. Caches need an eviction policy (LRU, LFU, FIFO) and care about what to cache (skip user-specific or time-sensitive queries). Warning: careless caching leaks data — caching a personalized answer to a seemingly generic question can serve one user's private result to another.
Seeing what your system is doing
Once it's built, you have to be able to watch it. Observability instruments the system so that when something breaks, its logs and metrics let you find out why without shipping new code — a stronger promise than plain "monitoring."
Key concept · The three pillars — metrics, logs, traces
Metrics (aggregated numbers) tell you something is wrong: design them around your failure modes (format-failure rate, factual consistency, toxicity, guardrail-trigger and refusal rates, plus latency/cost and length metrics), and correlate them to a north-star business metric (DAU, session duration). Logs (append-only event records) tell you what happened: the rule is log everything — configs, the final prompt, tool calls and outputs, component start/stop/crash — each tagged with IDs so you can trace it. Traces reconstruct a request's full execution path step by step, so a failed query can be pinpointed to the exact stage (bad retrieval vs. bad generation). Quality is judged by MTTD (mean time to detection), MTTR (mean time to response), and CFR (change failure rate) — and monitoring should feed back into your evaluation pipeline.
Key concept · Drift detection and orchestration
Because a system has many moving parts, watch for drift: system-prompt changes (a template edited upstream), user-behavior changes (users learn to write more concise prompts, shifting your metrics), and underlying-model changes (an API's model is updated without notice — versions can differ sharply in quality). Finally, an orchestrator wires components into an end-to-end pipeline in two steps — components definition (declare your models, data sources, tools) and chaining (function composition: process query → retrieve → build prompt → generate → evaluate → return or escalate). Tools include LangChain, LlamaIndex, Flowise, Langflow, and Haystack — but the advice is to start without an orchestrator, since it adds complexity and can obscure how your system works.
The fuel of the data flywheel
User feedback does two jobs — evaluating the application and informing its development — and in AI it matters even more, because feedback is proprietary data, and proprietary data is a competitive moat. A product that launches early gathers feedback that trains better models, feeding a data flywheel competitors struggle to catch. (Feedback is user data — respect privacy and let users know how it's used.)
Key concept · Explicit vs. implicit feedback
Explicit feedback is given in response to an explicit request: thumbs up/down, upvote/downvote, star rating, "Did we solve your problem?" It's well-understood but limited — there are only so many ways to ask. Implicit feedback is inferred from user actions, and is highly application-dependent: buying a recommended product signals a good recommendation; other signals include stopping a generation early, editing or regenerating a response, continuing the conversation, or correcting the model. Because both are captured, you can track cheap conversational metrics too (how often users stop generation halfway, average turns per conversation).
Key concept · Conversational signals
The conversational interface many AI apps use makes feedback natural: users encourage good behavior and correct mistakes the way they would in ordinary dialogue, and their language conveys feedback about both the application's performance and the user's preference. If you ask an assistant to book a Sydney hotel and then reply "actually, somewhere cheaper near the beach," that turn is rich feedback — even though the user never touched a thumbs-up button. Mining these signals is a distinctive opportunity of foundation-model applications.
Collecting feedback well — and its pitfalls
Feedback is only useful if you collect it thoughtfully and read it critically.
Key concept · Designing for feedback
Decide when and how to ask. Good feedback moments are in the natural flow of use — the initial experience (onboarding), points of failure (when a response is regenerated or a task is abandoned), and natural comparison points (offer two responses and let the user pick, which doubles as preference data). The design goal is to gather useful signal without degrading the experience: nagging for ratings annoys users, so lean on unobtrusive and implicit signals, and make giving feedback feel like a natural part of the interaction rather than an interruption.
Key concept · The limits — biases and degenerate feedback loops
Feedback is not ground truth. It carries biases: users may reward answers that merely sound confident or agreeable, favor certain positions, or give feedback unrepresentative of the silent majority (only the very happy or very angry respond). More dangerous are degenerate feedback loops: when a system optimizes on feedback it also shaped, it can spiral — a model tuned to please starts telling users what they want to hear, or a recommender that surfaces popular items makes them more popular still, narrowing behavior over time. Guard against it by monitoring for drift, keeping independent evaluation, and not treating raw feedback as an unbiased signal. Used carefully, feedback closes the loop that makes an AI product improve with every user — which is where the whole book has been heading.
Chapter 10 quiz
Fifteen questions on architecture, observability, and user feedback. Answer first, then expand Show answer.
Part A · Multiple choice
The chapter builds an architecture by:
- A Starting simple and adding components as needs arise
- B Building everything at once
- C Avoiding all components
- D Using only one model, always
Show answer
A. Query → model → response, then context, guardrails, routing, caching…
Input guardrails primarily protect against:
- A Leaking private information to external APIs and bad/malicious prompts
- B Slow hardware
- C Model size
- D Nothing
Show answer
A. E.g. PII redaction with a reversible map; jailbreak/injection defense.
Output-guardrail failures fall into which two categories?
- A Quality and security
- B Fast and slow
- C Cheap and expensive
- D Input and output
Show answer
A. Quality (malformatted, hallucinated) and security (toxic, leaked, brand-risk).
A model router is typically:
- A An intent classifier (or next-action predictor) that sends queries to the right solution
- B The largest model available
- C A database
- D A tokenizer
Show answer
A. Small, fast, cheap — can also detect out-of-scope or ambiguous queries.
A key benefit of a model gateway is:
- A Centralized access control, cost management, and fallback across models
- B A better model
- C Faster training
- D More parameters
Show answer
A. Update the gateway once when an API changes, not every app.
Semantic caching differs from exact caching in that it:
- A Reuses a result for a semantically similar query, not only an identical one
- B Never reuses results
- C Only caches images
- D Requires no similarity judgment
Show answer
A. More powerful but riskier — it depends on judging similarity correctly.
The three pillars of observability are:
- A Metrics, logs, and traces
- B CPU, GPU, RAM
- C Prompt, model, output
- D Speed, size, cost
Show answer
A. Metrics say something's wrong; logs say what; traces show the full path.
Which is implicit (not explicit) feedback?
- A A user regenerating or editing a response, or stopping generation early
- B A thumbs-up button
- C A star rating
- D Answering "Did we solve your problem?"
Show answer
A. Implicit feedback is inferred from actions.
Why is user feedback especially valuable for AI products?
- A It's proprietary data that powers a data flywheel — a competitive moat
- B It's always unbiased
- C It replaces evaluation
- D It reduces model size
Show answer
A. Early users → more data → better models → harder to catch up to.
Part B · True or false
Guardrails are free — they add no latency.
Show answer
False. There's a reliability-vs-latency trade-off; some teams skip guardrails to protect latency.
You should always start a project with a full orchestration framework.
Show answer
False. Start without one — orchestrators add complexity and can obscure how your system works.
An underlying model can change even when the API stays the same, causing drift.
Show answer
True. Providers may update models without notice; also watch for system-prompt and user-behavior drift.
Careless caching can leak one user's private answer to another user.
Show answer
True. Caching a personalized answer to a seemingly generic question is the classic trap.
Part C · Short answer
What is a degenerate feedback loop?
Show answer
When a system optimizes on feedback it also shaped, it can spiral — e.g. a model tuned to please starts telling users what they want to hear, or popular items get recommended more and become more popular, narrowing behavior over time.
What two things can a user's natural-language turn convey as feedback?
Show answer
The application's performance (was the response good?) and the user's preference (what they actually want) — e.g. "actually, somewhere cheaper near the beach."