Five levels, adopted progressively
Implementing every pattern at once is impractical. The key is progressive adoption: build a foundation of resilience, then add sophistication as the system grows in responsibility. The patterns organize into five levels.
| Level | Sophistication | Enabled patterns |
|---|---|---|
| 1 | Basic orchestration | None — happy path only; any failure is catastrophic. |
| 2 | Reactive recovery | Parallel Execution Consensus, Watchdog Timeout, Adaptive Retry. |
| 3 | Adaptive fault tolerance | Auto-Healing, Fallback Model, Incremental Checkpointing, Rate-Limited Invocation, Delayed Escalation. |
| 4 | Observable & auditable | Causal Dependency Graph, Trust Decay, Canary Agent Testing. |
| 5 | Self-governed & secure | Agent Mesh Defense, Execution Envelope Isolation, Majority Voting. |
The recommended rollout: start at Level 2 for baseline stability with minimal change, progress to Level 3 for recovery and efficiency, then add Levels 4–5 for the governance, security, and observability that critical production workloads require.
Safety nets so failure doesn't cascade
No component is perfect, so the first job is ensuring the failure or stall of one agent never becomes a system-wide outage.
| Pattern | What it does |
|---|---|
| Parallel Execution Consensus | Run the task on several agents in parallel and reconcile — cross-validate a result. |
| Majority Voting Across Agents | Multiple agents vote; the majority answer wins, filtering out individual errors. |
| Watchdog Timeout Supervisor | A supervisor kills and recovers a hanging/stalled agent before it blocks the system. |
| Adaptive Retry with Prompt Mutation | Retries a failed call while mutating the prompt to escape a repeating failure. |
| Auto-Healing Agent Resuscitation | Detects a dead agent and restarts/replaces it automatically. |
| Incremental Checkpointing | Saves progress so a long task resumes from the last checkpoint, not the start. |
| Delayed Escalation Strategy | Tiered escalation — try cheaper recovery first, escalate (e.g. to a human) only when needed. |
| Fallback Model Invocation | Switches to a backup model when the primary fails or is unavailable. |
| Rate-Limited Invocation | Throttles calls so the system stays a "good citizen" to its dependencies under load. |
Key concept · Performance is a feature of robustness
A system that fails under load is not robust. Optimizing for Translation Overhead keeps data handling between agents efficient, and Rate-Limited Invocation prevents an agent from overwhelming the tools and services it depends on — resilience is as much about staying performant as recovering from crashes.
Hardening, trust, and safe change
Security is a core tenet of robustness — a system must resist malicious attacks as well as accidental failure. Levels 4–5 add observability, trust management, and defense.
- Causal Dependency Graph
- Traces cause and effect across agent actions, creating a clear audit trail for debugging and accountability.
- Agent Self-Defense
- Hardens an individual agent against prompt injection and manipulation.
- Agent Mesh Defense
- System-level protection against a compromised agent spreading harm through the mesh.
- Execution Envelope Isolation
- Sandboxing — confines an agent's actions to a safe boundary.
- Trust Decay and Scoring
- Continuously scores agent reliability; trust decays after failures, gating future authority.
- Canary Agent Testing
- Validates updates on a small slice of traffic before full rollout.
Key concept · Layered, progressive resilience
Robustness is not all-or-nothing. Following the maturity model, start with simple reactive recovery and progressively adopt adaptive, then auditable, then secure patterns as systems grow. Consensus patterns (Parallel Execution, Majority Voting) and safety nets (Fallback Model, Watchdog) ensure one agent's failure doesn't cascade; security patterns (Causal Dependency Graph, Self-Defense, Mesh Defense) make the system resilient to attack.
Chapter 7 quiz
Fifteen questions on the robustness spectrum, recovery/consensus patterns, and security/observability patterns. Answer first, then expand Show answer.
Part A · Multiple choice
What does the robustness maturity model recommend?
- A Implement all 16 patterns at once
- B Progressive adoption — build a resilience foundation, add sophistication as the system grows
- C Skip recovery, go straight to security
- D Never add patterns
Show answer
B. Start at Level 2 (reactive recovery), progress to Level 3, then Levels 4–5 for governance and security.
Which pattern kills and recovers a hanging or stalled agent?
- A Majority Voting
- B Watchdog Timeout Supervisor
- C Trust Decay
- D Canary Agent Testing
Show answer
B — Watchdog Timeout Supervisor. It prevents a stalled agent from blocking the whole system.
Adaptive Retry with Prompt Mutation differs from a plain retry because it:
- A Retries forever
- B Mutates the prompt on each retry to escape a repeating failure
- C Switches models
- D Sandboxes the agent
Show answer
B. Changing the prompt avoids repeating the same failing path; switching models is Fallback Model Invocation.
Which pattern creates a clear audit trail of cause and effect across agent actions?
- A Causal Dependency Graph
- B Execution Envelope Isolation
- C Rate-Limited Invocation
- D Incremental Checkpointing
Show answer
A — Causal Dependency Graph. It supports observability and accountability at Level 4.
Which pattern hardens an individual agent against prompt injection?
- A Agent Mesh Defense
- B Agent Self-Defense
- C Majority Voting
- D Fallback Model
Show answer
B — Agent Self-Defense. Mesh Defense is the system-level protection against a compromised agent.
Execution Envelope Isolation is essentially:
- A Sandboxing an agent's actions within a safe boundary
- B A voting scheme
- C A checkpoint store
- D A retry loop
Show answer
A. It confines what an agent can do, containing damage from a compromised or misbehaving agent.
Canary Agent Testing is used to:
- A Validate updates on a small traffic slice before full rollout
- B Permanently replace all agents
- C Encrypt memory
- D Throttle API calls
Show answer
A. Safe, validated change management for agent updates.
Part B · True or false
A production-grade agent only needs to complete its task correctly; failure handling is optional.
Show answer
False. Robustness means architecting for the inevitability of failure, error, and unexpected conditions.
Fallback Model Invocation switches to a backup model when the primary fails or is unavailable.
Show answer
True. It's a Level-3 safety net that keeps the system running.
Rate-Limited Invocation keeps the system a "good citizen" to the dependencies it calls.
Show answer
True. Performance under load is treated as part of robustness.
Incremental Checkpointing forces a long task to restart from the beginning after a failure.
Show answer
False. It saves progress so the task resumes from the last checkpoint, not the start.
Part C · Short answer
Name the five levels of the robustness spectrum.
Show answer
1 Basic orchestration, 2 Reactive recovery, 3 Adaptive fault tolerance, 4 Observable & auditable, 5 Self-governed & secure.
Name two consensus/voting patterns that guard against a single agent's error.
Show answer
Parallel Execution Consensus and Majority Voting Across Agents.
Name the three security/defense patterns from Levels 4–5.
Show answer
Any three of: Causal Dependency Graph, Agent Self-Defense, Agent Mesh Defense, Execution Envelope Isolation, and Trust Decay & Scoring.
Which Level-2 patterns give a baseline of stability with minimal architectural change?
Show answer
Parallel Execution Consensus, Watchdog Timeout, and Adaptive Retry — the reactive-recovery starting point.