Five dimensions, one operating model
Monitoring an AI agent is not a single dashboard — it is a multilayered practice spanning five dimensions, run under an operating model with defined roles, standardized metric definitions, a log-review cadence, and change management.
| Dimension | What it watches |
|---|---|
| Operational health | Uptime/availability, error frequency, throttling, processing delays. |
| Performance | Response times, action success rates, tool-invocation reliability, completion. |
| Quality & accuracy | Appropriateness of outputs, alignment with business rules, deviation from expected behavior. |
| Usage insights | Volume trends, active-user adoption, feature utilization, behavioral patterns. |
| Risk, compliance & security | Guardrail violations, sensitive-data handling, suspicious spikes, policy adherence. |
Key concept · The monitoring toolset
- Azure Monitor — core telemetry, dashboards, alert rules, KQL diagnostics via Log Analytics.
- Microsoft 365 Admin Analytics — usage volume, adoption, engagement trends.
- Copilot & Agent dashboards — invocation frequency, completion trends, guardrail events.
- Power Platform Admin Center — environment health, connector usage, flow telemetry, DLP impact.
- Power BI custom dashboards — KPI views, usage heatmaps, drift visualizations, compliance trends.
Best practices: centralize logs, standardize naming, define clear SLAs, automate alerting on critical workflows, and fold monitoring output into monthly operational reviews.
Operational, quality, and user-centered
Performance monitoring groups metrics into three families, plus the model-level signals unique to generative AI.
- Operational
- Latency, throughput, error rate, resource/token utilization.
- Quality & reasoning
- Response accuracy, knowledge coverage, multi-step action effectiveness.
- User-centered
- Satisfaction indicators, abandonment rate, task-completion rate.
Key concept · Model-level signals
Generative agents add signals of their own: model drift (shifting response patterns, declining accuracy, more incorrect information / off-topic output), token consumption (the cost-to-performance ratio, thick vs thin prompting), and reliability indicators (latency jumps, changes in model-selection effectiveness, external-dependency errors).
Telemetry, interpreted
Telemetry falls into four categories — operational, model-level, behavioral, and governance/compliance. The architect's job is to read patterns across them and trace each to a root cause and a targeted fix.
| Signal | Possible cause | Architect action |
|---|---|---|
| Latency increase | Model overload, connector lag | Optimize workflow, cache data |
| High token usage | Verbose outputs, unclear prompts | Adjust prompt patterns |
| Error spike | Integration/API failure | Validate dependencies |
| Quality drop | Model drift, missing context | Update knowledge sources |
| Guardrail triggers | Policy conflict | Adjust governance rules |
Key concept · The diagnosis workflow
Six steps: monitor key metrics (establish baselines) → identify anomalies (deviations from normal) → correlate related signals → determine root cause (model-based, integration-based, or prompt-based) → apply targeted tuning → validate improvements by comparing before/after telemetry. Tuning levers: prompt refinement, knowledge updates, behavioral adjustments (fallback logic, orchestration), and cost optimization.
Business-aligned KPIs anchor the whole loop: responsiveness (median response time), accuracy & relevance, reliability (low failure rate), cost-effectiveness (balanced tokens/model choice), and user-outcome completion without manual intervention.
Module 8 quiz
Fourteen questions on the monitoring model and tools, the metric families, and telemetry interpretation and tuning. Answer first, then expand Show answer.
Part A · Multiple choice
Which is NOT one of the five monitoring dimensions for AI agents?
- A Operational health
- B Quality & accuracy
- C Marketing reach
- D Risk, compliance & security
Show answer
C — Marketing reach. The five are operational health, performance, quality & accuracy, usage insights, and risk/compliance/security.
Which tool provides core telemetry, alert rules, and KQL-based diagnostics via Log Analytics?
- A Azure Monitor
- B Microsoft 365 Admin Analytics
- C Power BI
- D Copilot Studio test canvas
Show answer
A — Azure Monitor. It integrates with Log Analytics Workspaces for KQL diagnostics on latency, throughput, and connector failures.
A steady rise in incorrect or off-topic responses over time most likely indicates:
- A Model drift
- B A DLP policy update
- C Better prompting
- D Lower token usage
Show answer
A — Model drift. Shifting response patterns and declining accuracy on recurring tasks are drift indicators; refresh grounding and re-evaluate.
Per the signal map, high token usage is best addressed by:
- A Validating dependencies
- B Adjusting prompt patterns
- C Adding more connectors
- D Increasing model size
Show answer
B. High token usage usually comes from verbose outputs or unclear prompts — tighten the prompt patterns.
In the six-step diagnosis workflow, what comes immediately after "determine root cause"?
- A Identify anomalies
- B Apply targeted tuning
- C Monitor key metrics
- D Retire the agent
Show answer
B — Apply targeted tuning, then validate improvements against before/after telemetry.
Which is a tuning lever the module recommends?
- A Prompt refinement
- B Deleting the audit logs
- C Disabling monitoring
- D Removing all guardrails
Show answer
A — Prompt refinement. Along with knowledge updates, behavioral adjustments, and cost optimization.
Part B · True or false
Reacting to individual telemetry events is more valuable than recognizing patterns across signals.
Show answer
False. The core principle is to focus on patterns, not isolated events — correlate signals before acting.
An effective monitoring operating model needs defined roles, standardized metric definitions, and a log-review cadence.
Show answer
True. Plus incident-response workflows, change management, and documented expected behaviors.
Behavioral telemetry includes user interaction logs, conversation transcripts, and feature-usage signals.
Show answer
True. It complements operational telemetry (logs, traces, exceptions) and model-level telemetry.
Guardrail-violation counts belong in the risk/compliance/security monitoring dimension.
Show answer
True. Along with sensitive-data handling, suspicious spikes, and policy adherence.
Part C · Short answer
Name the three metric families in agent performance monitoring.
Show answer
Operational, quality & reasoning, and user-centered metrics.
List the four categories of telemetry the architect interprets.
Show answer
Operational, model-level, behavioral, and governance/compliance.
Give three business-aligned KPIs for an AI system.
Show answer
Any three of: responsiveness, accuracy & relevance, reliability, cost-effectiveness, and user-outcome completion.
Name three monitoring best practices from the module.
Show answer
Any three of: centralize logs, standardize naming conventions, define clear SLAs, automate alerting, and integrate outputs into operational reviews.