Start simple, scale intelligently
The strategic path to a sophisticated agent begins with the Single Agent Baseline — the minimal working agent — and layers on more capability only as responsibilities demand it. Each pattern is a deliberate architectural choice, not an accident.
Single Agent Baseline
The starting point: an LLM core with basic tool use, before any of the advanced capabilities are added.
Agent-Specific Context & Memory
Short-term memory (a sliding window of recent turns, or a running summary in the context window) plus long-term memory (a vector database or key-value store).
Retrieval-Augmented Generation
Grounds the agent in verifiable facts by fetching relevant external knowledge at inference time.
Structured Reasoning & Self-Correction
Chain-of-Thought for a transparent reasoning pass, plus a self-critique step where an internal auditor reviews the answer before returning it.
Multimodal Sensory Input
Processing visual documents and UIs — the leap from a text processor to a true co-pilot.
Remembering and staying factual
Two capabilities give an agent continuity and reliability. Memory comes in two tiers, and RAG keeps outputs grounded.
- Short-term memory
- Held in the LLM's context window — a sliding window of the last N turns, or a summarized running state.
- Long-term memory
- Persistent storage — a vector database or key-value store, often reached through a RAG system.
Key concept · Grounding and reliability are paramount
For an agent to be trusted in an enterprise context it must be grounded in facts (RAG) and its reasoning must be robust and transparent (self-correction). Memory provides continuity across a conversation; RAG connects the agent to an authoritative knowledge base — and long-term memory is often implemented as a RAG system over stored interactions.
Thinking transparently, then seeing
Structured Reasoning & Self-Correction makes an agent's thinking both more reliable and more auditable. Rather than emitting an immediate answer, the agent reasons in steps and then checks itself.
Key concept · Chain-of-Thought + self-critique
Use Chain-of-Thought (CoT) for an initial reasoning pass that produces a transparent, step-by-step trace. Then add self-critique: before returning, the agent's internal auditor is prompted to review its own answer for errors or missed constraints and revise if needed. Together they keep the agent on task and make its conclusions inspectable.
Multimodal Sensory Input is the next frontier: giving an agent the ability to process visual information — documents, screenshots, user interfaces — elevates it from a text-based processor to a true co-pilot capable of operating in human-centric, visually-driven workflows. Success across all these patterns is measured with pattern-specific evaluation metrics during enterprise rollout.
Chapter 9 quiz
Fourteen questions on the incremental capability stack, memory and RAG, structured reasoning, and multimodal input. Answer first, then expand Show answer.
Part A · Multiple choice
What is the recommended starting point when building a capable agent?
- A A full multimodal multi-agent system
- B The Single Agent Baseline, then layer on capabilities
- C Fine-tuning first
- D A vector database
Show answer
B. Start simple and scale intelligently — add memory, RAG, and reasoning as responsibilities grow.
Where is short-term memory typically held?
- A In a vector database
- B Within the LLM's context window (sliding window or summary)
- C On disk permanently
- D In a proxy agent
Show answer
B. A sliding window of recent turns or a running summary; long-term memory uses persistent storage.
Long-term memory is commonly implemented with:
- A A sliding window
- B A vector database or key-value store (often via RAG)
- C The system prompt only
- D A watchdog timer
Show answer
B. Persistent storage reached through a retrieval system.
What does the RAG pattern give an individual agent?
- A Faster token generation
- B Grounding in verifiable facts from external knowledge
- C Multimodal perception
- D A security boundary
Show answer
B. RAG fetches relevant external knowledge to keep outputs factual.
Structured Reasoning & Self-Correction combines which two techniques?
- A Chain-of-Thought and a self-critique step
- B Fine-tuning and quantization
- C Sandboxing and firewalls
- D Sliding window and summarization
Show answer
A. CoT for a transparent reasoning pass, then an internal auditor reviews the answer before returning it.
The Multimodal Sensory Input pattern elevates an agent by letting it:
- A Run faster
- B Process visual documents and UIs, becoming a true co-pilot
- C Skip grounding
- D Avoid memory
Show answer
B. Visual perception moves it beyond a text-based processor.
Part B · True or false
A capable agent emerges automatically from a powerful LLM without deliberate design.
Show answer
False. Agents are built, not summoned — capability is the result of deliberate architectural choices.
Grounding in facts and transparent reasoning are essential for enterprise trust.
Show answer
True. RAG provides grounding; self-correction makes reasoning robust and transparent.
A simple short-term memory strategy is a "sliding window" of the last N conversation turns.
Show answer
True. Summarization of the running state is the other common approach.
Self-critique means an external human always reviews every answer.
Show answer
False. The agent's own internal auditor reviews the answer before returning it — an automated self-check.
Part C · Short answer
Name the five agent-level patterns in build order.
Show answer
Single Agent Baseline → Agent-Specific Context & Memory → RAG → Structured Reasoning & Self-Correction → Multimodal Sensory Input.
Distinguish short-term from long-term memory.
Show answer
Short-term lives in the context window (sliding window/summary); long-term is persistent storage (vector DB / key-value), often via RAG.
What are the two roles Chain-of-Thought and self-critique play together?
Show answer
CoT gives a transparent, step-by-step reasoning pass; self-critique reviews and corrects the answer before it's returned.
Why is multimodal input called "the next frontier" for agents?
Show answer
Because processing visual information (documents, UIs) turns a text-only processor into a co-pilot able to operate in human-centric workflows.