when your model slowly gets worse but all the dashboards say it’s fine. silent failure as a service.
Degradation in model performance over time as input data diverges from training data. Drift detection is a core S4 control. Undetected drift means increasingly unreliable decisions while the system appears functional. The model does not change — the world does. The gap between what the model learned and what it encounters grows wider with every passing day, and without monitoring, that gap is invisible.
Why It Matters
Model drift is the most common mode of AI system failure and the least dramatic. There is no crash, no error message, no incident alert. The model continues to operate within its technical parameters — latency is fine, throughput is fine, the API returns 200 — while its outputs become progressively less accurate. Research indicates that 91% of machine learning models degrade over time due to drift. This is not an edge case. It is the baseline expectation.
The danger is compounded by two factors. First, drift is gradual. A model that was 95% accurate at deployment does not drop to 50% overnight. It might drop to 93%, then 90%, then 87% — each increment too small to trigger investigation, the cumulative effect large enough to invalidate decision-making. Second, most monitoring systems track operational metrics (latency, error rates, uptime) rather than performance metrics (accuracy, precision, recall against current ground truth). The dashboards stay green while the decisions get worse.
For regulated environments, the implications are severe. The EU AI Act requires high-risk system providers to implement post-market monitoring (Art. 72) that specifically covers performance degradation. A system that was compliant at deployment can become non-compliant through drift alone — without any code change, configuration change, or human intervention.
The Stress Test
Your credit scoring model was deployed eighteen months ago. It passed all validation tests. The model risk management team signed off. The documentation is complete. The model scores 50,000 applications per month.
A regulatory examination asks for evidence of ongoing performance monitoring. You provide operational dashboards: uptime 99.97%, average latency 23ms, zero system errors. The examiner asks for performance metrics: current accuracy, false positive rate, false negative rate measured against recent outcomes. You do not have them. The model was validated at deployment. It has not been re-evaluated since. You have eighteen months of production decisions with no evidence of continued accuracy. The examiner writes: “No post-market monitoring of model performance. Finding: material.”
In the Wild
The COVID-19 pandemic created the largest simultaneous drift event in the history of machine learning. Credit scoring models trained on pre-pandemic data encountered borrowers whose behaviour had fundamentally changed: mass unemployment, government stimulus payments, mortgage forbearance programmes, and altered spending patterns. Fraud detection models saw transaction patterns shift overnight as in-person commerce collapsed and digital payments surged.
Banks that had drift detection mechanisms identified the divergence within weeks and could respond — recalibrating thresholds, adding manual review layers, or temporarily switching to rules-based systems. Banks without drift monitoring continued running pre-pandemic models on post-pandemic data, producing scores that bore no relationship to actual risk. Some institutions reported that fraud losses doubled before the drift was identified.
The pandemic did not break the models. It broke the assumption that the future would resemble the past. Drift monitoring is the control that detects when that assumption fails.
Epic Systems’ sepsis prediction model, deployed across hundreds of US hospitals, was evaluated by researchers at the University of Michigan. The external validation found that the model’s performance had degraded significantly from its original reported metrics. The model generated a high number of false positives (alert fatigue) while missing actual sepsis cases at rates far exceeding the published benchmarks.
The drift had multiple sources: changes in clinical practice, shifts in patient demographics, and evolving documentation patterns in electronic health records. The model was trained on historical data that no longer reflected current clinical reality. Clinicians, overwhelmed by false alarms, had begun ignoring the model’s alerts entirely — a behavioural drift that compounded the statistical drift.
A model that cries wolf too often trains its users to ignore it. When it finally gets it right, nobody is listening. Drift does not just degrade the model. It degrades trust in the system.
A major European e-commerce platform discovered that its recommendation engine was systematically suggesting discontinued and out-of-stock products. The model had been trained on historical purchase data and was accurately predicting what customers would have bought based on past patterns. However, the product catalogue had evolved: new brands had been added, popular items had been discontinued, and seasonal collections had rotated.
The model’s accuracy metrics, measured against historical test data, remained stable. But the practical value of the recommendations had collapsed because the model was optimising for a product catalogue that no longer existed. Revenue from recommendation-driven purchases dropped 34% over six months before the drift was identified.
The model was not wrong about what customers wanted. It was wrong about what was available. Drift is not always about the model getting dumber. Sometimes the world just moves on.
How to Govern It
If you are not measuring model performance against current reality, you are not monitoring. You are hoping.
Within the AI Control Index, model drift governance spans multiple layers and shields:
- Observability (S4) — The primary control layer. Drift detection requires statistical monitoring of input distributions (data drift), output distributions (prediction drift), and performance metrics against ground truth (concept drift). Detection must be automated and continuous, not periodic and manual.
- AI Engineering (L5) — Drift response requires predefined playbooks: retraining triggers, recalibration procedures, fallback models, and retirement criteria. The engineering team must have a response plan before drift occurs, not after.
- GRC (S1) — Evidence Factory captures drift monitoring results, threshold breach records, and remediation actions as governance artifacts. This is the evidence a regulator will request under post-market monitoring obligations.
- Applications & Agents (L4) — Circuit breakers that trigger when drift exceeds acceptable bounds: automatic escalation to human review, output confidence warnings, or system suspension pending investigation.
- Ethics & Fairness (L2) — Differential drift monitoring across protected groups. A model can drift uniformly (performance degrades for everyone) or asymmetrically (performance degrades faster for specific demographics). The latter is a fairness incident, not just a performance incident.
When It’s Relevant
Every AI model in production. Without exception. The EU AI Act (Art. 72) requires post-market monitoring for high-risk systems. The US Federal Reserve’s SR 11-7 requires ongoing monitoring of model performance. The NIST AI RMF maps drift to Measure 2.6 and Manage 2.2. Every regulatory framework that addresses AI in production addresses drift.
Drift risk is highest when:
- The model operates in a fast-changing domain (financial markets, social media, fraud patterns)
- External events change the underlying data distribution (regulatory changes, economic shifts, pandemics)
- The model has not been re-evaluated against current ground truth in more than three months
- The monitoring system tracks operational metrics but not performance metrics
- The organisation has no predefined drift response playbook
Related Terms
References
- [1] Lu, J., Liu, A., Dong, F., Gu, F., Gama, J. and Zhang, G. (2019) ‘Learning under Concept Drift: A Review’, IEEE Transactions on Knowledge and Data Engineering, 31(12), pp. 2346–2363. doi: 10.1109/TKDE.2018.2876857.
- [2] Wong, A., Otles, E., Donnelly, J.P., Krber, A., Levine, D., et al. (2021) ‘External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients’, JAMA Internal Medicine, 181(8), pp. 1065–1070. doi: 10.1001/jamainternmed.2021.2626.
- [3] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act), Article 72: Post-market monitoring by providers. Official Journal of the European Union.
- [4] Board of Governors of the Federal Reserve System (2011) Supervisory Guidance on Model Risk Management (SR 11-7). Washington, D.C.
- [5] NIST (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology. Measure 2.6, Manage 2.2.
- [6] Bayram, F., Ahmed, B.S., Kassler, A. (2022) ‘From concept drift to model degradation: An overview on performance-aware drift detectors’, Knowledge-Based Systems, 245, 108632. doi: 10.1016/j.knosys.2022.108632.
- [7] Gama, J., Medas, P., Castillo, G. and Rodrigues, P. (2004) ‘Learning with drift detection’, Proceedings of the 17th Brazilian Symposium on Artificial Intelligence (SBIA), Lecture Notes in Computer Science, vol. 3171, pp. 286–295.
AI Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0
By Jeroen Janssen, Apparens