A plan, four test types, a lifecycle
Testing starts with a test plan — scope (features, workflows, channels, scenarios), representative test data, roles (who executes, validates, documents), and measurable success criteria for accuracy, speed, safety, and usability. From there, four test types cover the surface.
| Type | What it validates |
|---|---|
| Scenario-based | Real business workflows, ambiguous/incomplete inputs, multi-turn reasoning and memory. |
| Performance & reliability | High request volume, long interactions, complex multi-step tasks, concurrent sessions. |
| Safety & compliance | Sensitive-data protection, role-based access, policy/DLP triggers, rejection of disallowed instructions. |
| Usability | Clarity and helpfulness; whether users can prompt effectively without excessive refinement. |
Key concept · The nine-stage testing lifecycle
Test planning → scenario design → execution → measurement → analysis → tuning → re-test → approval (governance checkpoint) → deployment. Maintain a centralized log of results for cross-release comparison, automate repeatable scripts where possible, and gate every deployment on a governance checkpoint.
Six dimensions, with thresholds
Custom models must be validated across six dimensions — performance, quality & accuracy, safety & compliance, cost efficiency, operational reliability, and user experience — combining quantitative metrics (measurable, repeatable) with qualitative criteria (relevance, reasoning consistency, grounding integrity). Crucially, set success thresholds before deployment.
| Area | Metric | Success threshold |
|---|---|---|
| Performance | Latency | < 2 seconds |
| Quality | Accuracy | ≥ 90% correctness |
| Quality | Incorrect-information rate | ≤ 3% |
| Safety | Guardrail violations | 0 |
| Safety | Sensitive-output detection | 100% blocked |
| User experience | Satisfaction score | ≥ 4.5 / 5 |
Key concept · Quantitative vs qualitative
Numbers catch what's measurable — accuracy, latency, throughput, error rate, token efficiency, drift. Qualitative criteria catch what they miss — relevance & completeness, consistency of reasoning, grounding integrity (approved sources only), and UX quality. Validate continuously — before, during, and after deployment — and treat safety/compliance as baseline, not an afterthought.
Testing across multiple Dynamics 365 apps
AI-driven processes pull data and triggers from several systems, so testing must validate the entire business process — Order-to-Cash (Sales → Finance → Customer Service) or Case-to-Resolution (Customer Service → Field Service) — not individual modules. Workflow orchestration breaks when one app changes, and AI output quality depends on consistent, well-timed input data.
Key concept · An E2E scenario, structured
Every E2E test carries preconditions (data setup, environment readiness), step-by-step execution, the AI action/prediction trigger, cross-app data movement validation, post-condition checks, and exception/negative paths. Example: preload data → create Lead in Sales → AI surfaces opportunity → Sales Order → syncs to Finance → AI validates credit risk → fulfillment in Supply Chain → AI monitors delivery exceptions → approval → validate outputs across all apps.
A cross-app readiness checklist gates the test: confirm entity mappings, validate data-refresh timing/latency, verify business-process dependencies between apps, and ensure the AI uses the correct grounding sources. Best practices: test realistic high-value processes, validate with multiple phrasing variations, cover structured and unstructured data, run normal and stress loads, include exception paths and regulatory constraints, and document cases in repeatable templates for regression.
Module 9 quiz
Fourteen questions on the testing framework and test types, six-dimension validation with thresholds, and end-to-end cross-app scenarios. Answer first, then expand Show answer.
Part A · Multiple choice
Which test type validates that the agent rejects disallowed instructions and respects role-based access?
- A Usability testing
- B Safety & compliance testing
- C Performance testing
- D Scenario-based testing
Show answer
B — Safety & compliance. It confirms sensitive-data protection, RBAC, policy/DLP triggers, and rejection of disallowed instructions.
What comes immediately before "deployment" in the nine-stage testing lifecycle?
- A Execution
- B Approval (governance checkpoint)
- C Scenario design
- D Measurement
Show answer
B — Approval. The lifecycle ends: … re-test → approval (governance checkpoint) → deployment.
Per the example thresholds, the target for guardrail violations is:
- A ≤ 3%
- B 0
- C ≥ 90%
- D < 2 seconds
Show answer
B — 0. And sensitive-output detection should be 100% blocked — safety is a baseline requirement.
Which is a purely qualitative validation criterion (not captured by a number)?
- A Latency
- B Throughput
- C Grounding integrity
- D Token efficiency
Show answer
C — Grounding integrity. Whether the model uses approved organizational knowledge exclusively — a qualitative check.
Why must AI solutions across multiple Dynamics 365 apps be tested end-to-end rather than module-by-module?
- A To reduce documentation
- B Because AI decisions depend on data across apps and orchestration breaks when one app changes
- C Because modules never fail
- D To avoid governance checkpoints
Show answer
B. Cross-app data sync, transformation, and timing all affect AI output quality — the whole process must be validated.
Which belongs in the cross-app readiness checklist before an E2E test?
- A Confirm entity mappings and data-refresh timing
- B Delete the test environment
- C Skip exception paths
- D Use one phrasing only
Show answer
A. Plus verifying business-process dependencies and correct grounding sources.
Part B · True or false
Success thresholds should be defined before deployment, not after results come in.
Show answer
True. Establish thresholds for each metric up front so results can be judged objectively.
Validation should happen only once, right before go-live.
Show answer
False. Validate continuously — before, during, and after production deployment.
Scenario-based testing should include ambiguous and incomplete user inputs.
Show answer
True. It reflects real usage and validates multi-turn reasoning, memory, and follow-up behavior.
Test cases should be documented in repeatable templates to enable regression testing.
Show answer
True. Repeatability and a centralized results log let you compare across releases.
Part C · Short answer
Name the four AI-agent test types.
Show answer
Scenario-based, performance & reliability, safety & compliance, and usability.
List the six validation dimensions for custom AI models.
Show answer
Performance, quality & accuracy, safety & compliance, cost efficiency, operational reliability, and user experience.
Name the six structural parts of an end-to-end test scenario.
Show answer
Preconditions, step-by-step execution, the AI action/prediction trigger, cross-app data-movement validation, post-condition validation, and exception/negative paths.
What four things should a complete agent test plan specify?
Show answer
Test scope, test data, test roles, and success criteria (measurable thresholds for accuracy, speed, safety, and usability).