Use a small dashboard with an owner and a response for each alert:
| Layer | Watch | Example response |
|---|---|---|
| Data | Freshness, availability violations, joins, missingness and label completeness | Repair upstream job; suspend affected decisions |
| Model | Mature-cohort calibration, capacity metrics or demand errors; supported population | Diagnose drift; recalibrate or evaluate challenger |
| Policy | No-action/fallback rates, constraints, budget, price movement, fatigue | Revert policy configuration or stop rollout |
| Service | p95 latency, timeout/error rate, duplicate deliveries, cost per decision | Route to baseline and repair serving |
| Business | Experiment contribution, returns, opt-outs, complaints and retention horizon | Check telemetry and stop according to declared guardrails |
Record baseline levels and set alert limits from business tolerance and observed variability. Tutorial thresholds are placeholders. Keep a suitable holdout or control so the model is not credited with changes in the world. Monitor subgroup performance and allocation against the project's customer policy.
Retrain on a documented schedule or justified trigger after checking the data. Compare the challenger on recent mature cohorts and a stable benchmark, then shadow or canary it. Update calibration and policy evaluation with the exact released model. Roll back the whole compatible release or switch to the baseline, and keep the incident records.