Everything is a promotable artifact
ALM for AI treats datasets, evaluation sets, grounding knowledge, prompt assets, policies/guardrails, and run telemetry as formal artifacts that carry versions and require promotion approval. They move through isolated environments under least-privilege access.
| Environment | Purpose | Solution type |
|---|---|---|
| Development | Build and iterate on agents, connectors, actions | Unmanaged |
| Test (UAT/QA) | Validate behavior, evaluate prompts, regression checks | Managed |
| Production | Stable, approved, monitored implementation | Managed |
Key concept · Red vs gold data
Red data is mutable and experimental (Dev/Test); gold data is frozen, promoted, and immutable (Pre-Prod/Prod). Never edit directly in production, use managed solutions only in Test and Prod, enforce role-based access at each tier, and require promotion gates backed by evidence — quality reports, bias checks, lineage, and security sign-off.
Seven phases, gate by gate
Data powering AI moves through a seven-phase governance framework, each phase ending in a gate that must produce evidence before promotion.
Plan & catalog
Define data contracts (purpose, fields, retention, owners), classify sensitivity, register in the catalog, set KPIs and risk posture.
Ingest & prepare
Profile and remediate quality, generate lineage, stamp schema versions and hashes, build curated/feature sets via deterministic pipelines.
Develop & evaluate
Train on Dev/Test data only (never on production knowledge); run accuracy/safety/robustness/cost evaluation suites; store model/data cards.
Stage & approve
Privacy/security/compliance reviews, canary runs on masked Prod-like data, freeze gold datasets, sign immutability attestations, CAB approval.
Deploy & serve
Promote gold corpora/indexes, refresh retrieval stores, enforce region/residency and connector allow-lists, register the release.
Operate & monitor
Track latency, cost/token, success rate, safety violations, access denials; detect drift; trigger circuit breakers / HITL; re-evaluate on golden sets.
Evolve & retire
Retrain on updated gold sets and retest; apply retention/deletion to expired snapshots; preserve audit trails and lineage.
Key concept · Residency decision & RACI
For regulated workloads, default to in-region processing; enable cross-region overflow only under explicit admin approval, otherwise block or defer. Ownership is assigned by a RACI matrix — the data owner is accountable for the data contract and retention, the AI architect for evaluation thresholds and deployment, and security/compliance for residency configuration.
ALM for what runs on the data
The same discipline extends to Copilot Studio agents, connectors, and actions: build in Dev (unmanaged), validate in Test with DLP applied, publish to Prod as managed solutions only after security review. Export solution packages to source control, run monthly/sprint releases with an emergency-patch path, and keep a rollback plan.
- Connector release flow
- Author → validate auth → apply DLP → approve security → publish to Prod.
- Action lifecycle
- Design → build → validate → promote (Dev→Test→Prod) → monitor.
Key concept · Custom-model ALM (MLOps)
Custom models add drift-specific rigor across seven stages (plan & design → data prep → development → evaluation & approval → deployment → monitor & optimize → retirement), on four foundations — consistency, compliance, repeatability, operational readiness. Every promotion produces a Model Card (metrics, constraints, usage boundaries); use model registries with version locking and rollback paths, keep metadata (hyperparameters, training-data versions, environment snapshots), and trigger retraining on data drift or model drift. Gate 1 (Dev→Test) checks data quality, safety/bias, documentation; Gate 2 (Test→Prod) checks performance vs baseline, cost/latency, security/compliance approvals, and a confirmed rollback plan.
Module 10 quiz
Fifteen questions on environments and red/gold data, the seven-phase data ALM and its gates, and ALM for agents and custom models. Answer first, then expand Show answer.
Part A · Multiple choice
In the Copilot Studio environment strategy, which environments use managed solutions?
- A Dev only
- B Test and Production
- C Production only
- D None
Show answer
B — Test and Production. Dev uses unmanaged solutions for building; managed solutions are used in Test and Prod.
What best describes "gold" data?
- A Mutable, experimental data used in Dev
- B Frozen, promoted, immutable data used in Pre-Prod/Prod
- C Data that is never versioned
- D Data stored only in memory
Show answer
B. Gold data is immutable and version-locked; red data is the mutable experimental data in Dev/Test.
Which phase of the seven-phase data ALM freezes gold datasets and obtains CAB approval?
- A Plan & catalog
- B Ingest & prepare
- C Stage & approve
- D Evolve & retire
Show answer
C — Stage & approve. It runs compliance reviews and canary runs, freezes gold datasets, and secures Change Advisory Board approval.
For a regulated workload with no in-region capacity, the residency decision default is to:
- A Always process cross-region automatically
- B Block or defer unless overflow is explicitly approved under admin control
- C Ignore residency rules
- D Delete the workload
Show answer
B. Default to in-region; cross-region overflow requires explicit admin approval, otherwise block or defer.
What artifact is produced at each custom-model promotion, documenting metrics, constraints, and usage boundaries?
- A A Model Card
- B A DLP policy
- C A connector manifest
- D A RACI chart
Show answer
A — Model Card. Along with data contracts and risk assessments as core ALM artifacts.
Which is the correct connector release flow?
- A Publish → author → validate → approve
- B Author → validate auth → apply DLP → approve security → publish to Prod
- C Approve → publish → author → validate
- D Publish to Prod → apply DLP
Show answer
B. Connectors are built in Dev, DLP applied in Test, and published to Prod only after security review.
Custom-model Gate 2 (Test→Prod) requires which of the following?
- A Only that the model exists
- B Performance above baseline, acceptable cost/latency, security/compliance approvals, and a confirmed rollback plan
- C Removal of all monitoring
- D Training on production knowledge
Show answer
B. Gate 1 covers data quality/safety/documentation; Gate 2 covers performance, cost, approvals, and rollback.
Part B · True or false
Models may be trained directly on production knowledge sources during the Develop & evaluate phase.
Show answer
False. Train on Dev/Test data only — never on production knowledge.
Direct editing in the production environment is an accepted ALM practice.
Show answer
False. No direct editing in production; changes flow Dev → Test → Prod through managed solutions.
Version locking ensures a specific model version is immutable in production.
Show answer
True. Model registries use version locking plus rollback paths for production safety.
Audit trails and lineage should be preserved even when data snapshots are retired.
Show answer
True. The Evolve & retire phase applies retention/deletion but preserves audit trails and lineage for compliance.
Part C · Short answer
Name the seven phases of the data ALM framework.
Show answer
Plan & catalog → Ingest & prepare → Develop & evaluate → Stage & approve → Deploy & serve → Operate & monitor → Evolve & retire.
Name the four foundations a strong custom-model ALM process ensures.
Show answer
Consistency, compliance, repeatability, and operational readiness.
Distinguish data drift from model drift.
Show answer
Data drift = changes in the input data distribution; model drift = output deviating from expected performance due to environmental or data changes. Either can trigger retraining.
What is a "data contract" in AI ALM?
Show answer
A formal agreement defining a dataset's purpose, fields, retention period, owners, and usage boundaries — established in the Plan & catalog phase.