the part of AI governance that happens after you ship, which is also the part most teams forget exists.
A structured system required by EU AI Act Art. 72 for high-risk AI systems after deployment. Covers telemetry collection, complaint intake, near-miss logging, serious incident triage (Art. 73), and regulator notification. The system must be proportionate to the nature of the AI technology and the risks of the high-risk AI system, and must be established before the system is placed on the market. Post-market monitoring is not optional post-deployment hygiene — it is a legal requirement with specific operational components.
Why It Matters
Most AI governance effort concentrates on pre-deployment: risk assessment, testing, evaluation, approval. Post-market monitoring is where that governance must prove it was not theatre. An AI system that passed every pre-deployment gate can still fail in production — because the data distribution shifts, because users interact with the system in unexpected ways, because the world changes and the model does not.
The EU AI Act recognises this explicitly. Article 72 requires providers of high-risk AI systems to establish a post-market monitoring system that “actively and systematically” collects, documents, and analyses data on the performance of the system throughout its lifetime. Article 73 adds serious incident reporting: providers and deployers must report incidents to market surveillance authorities within 15 days. The regulatory architecture assumes that things will go wrong after deployment and requires infrastructure to detect and respond when they do.
The operational reality lags far behind the requirement. A 2025 Stanford HAI study found that only 23% of organisations deploying AI had structured post-deployment monitoring that went beyond basic infrastructure metrics (uptime, latency, error rates). The remaining 77% were monitoring whether the system was running but not whether it was working correctly. The distinction matters: a model can have 99.9% uptime while delivering increasingly inaccurate outputs to every user.
The Stress Test
A market surveillance authority asks you to demonstrate your post-market monitoring system for a high-risk AI deployment. You show them your observability dashboard: uptime, latency, throughput, error rates. They ask where the output quality monitoring is. You explain that the model was extensively tested before deployment. They ask for the post-deployment performance data. You have nine months of infrastructure telemetry and zero months of output quality measurement.
They then ask about your serious incident reporting process under Article 73. You describe an internal incident management system. They ask how many near-misses were logged in the last quarter. The answer is zero — not because there were no near-misses, but because nobody defined what a near-miss looks like for this system and nobody was looking. The absence of incidents in your log is not evidence of safety. It is evidence that the detection mechanism does not exist.
In the Wild
Research published in Science by Obermeyer et al. revealed that a healthcare algorithm used by UnitedHealth Group to identify patients needing extra care was systematically discriminating against Black patients. The algorithm used healthcare costs as a proxy for health needs. Because Black patients historically had less access to healthcare (and therefore lower costs), the algorithm concluded they were healthier. At a given risk score, Black patients were considerably sicker than White patients. The bias was not present in the training data in an obvious way — it emerged from the proxy variable and was amplified over time as the model’s recommendations influenced subsequent care allocation.
The algorithm was not monitored for fairness drift after deployment. It ran for years. The bias was discovered by external researchers, not by the deployer’s monitoring system — because there was no fairness monitoring system.
Zillow’s AI-powered home-buying programme, Zillow Offers, used machine learning models to estimate home values and make instant purchase offers. In Q3 2021, the models began systematically overvaluing homes. The drift was not detected by monitoring systems because the models were evaluated against their own predictions rather than against actual market outcomes. Zillow purchased approximately 27,000 homes at above-market prices. The company wrote down $881 million, laid off 25% of its workforce, and shut down the programme entirely.
Zillow monitored whether its models were consistent. It did not monitor whether they were correct. The models were consistently wrong. For months.
The U.S. Food and Drug Administration issued its final guidance on predetermined change control plans for AI-enabled medical devices (Software as a Medical Device). The guidance requires manufacturers to define, before deployment, the specific modifications the AI system might undergo and the methodology for evaluating whether those changes maintain safety and effectiveness. The framework requires continuous real-world performance monitoring with predefined triggers for re-evaluation. The FDA explicitly rejected the model where AI devices are approved once and left unmonitored.
The FDA did not wait for the EU AI Act. It built post-market monitoring requirements into the approval process itself. The approval is conditional on monitoring. No monitoring, no approval.
How to Govern It
Post-market monitoring is not a dashboard. It is a system with defined inputs, triggers, escalation paths, and evidence outputs.
Within the AI Control Index, post-market monitoring governance spans Observability, Evidence, and incident management:
- Observability (S4) — Runtime telemetry that goes beyond infrastructure metrics to include output quality monitoring, fairness drift detection, confidence score distribution tracking, and factual accuracy sampling. If you are only monitoring uptime, you are monitoring the container, not the AI.
- Circuit Breakers — Predefined thresholds that trigger automatic intervention when monitoring detects degradation beyond acceptable limits. A drift threshold breach should trigger a circuit breaker, not an email to a committee.
- Near-Miss Logging — A structured taxonomy of near-miss events specific to the AI system, with defined detection criteria and logging requirements. Near-misses are leading indicators. Serious incidents are lagging indicators. The monitoring system must capture both.
- Serious Incident Triage (Art. 73) — A documented process for evaluating whether a detected event meets the threshold for serious incident reporting. Includes severity classification, root cause analysis, and the 15-day reporting timeline to market surveillance authorities.
- Evidence Factory (S1) — All monitoring outputs — telemetry records, drift alerts, near-miss logs, incident reports, remediation actions — are captured as governance artifacts with timestamps, ownership, and audit trails.
When It's Relevant
Every AI system in production. The EU AI Act (Art. 72) mandates post-market monitoring for high-risk systems, but the operational principle applies universally: any AI system that interacts with users, makes decisions, or generates outputs can degrade after deployment. The question is not whether to monitor. The question is whether you will detect degradation before or after it causes harm.
Post-market monitoring is most critical when:
- The AI system operates on data distributions that change over time (nearly all real-world systems)
- The system’s outputs feed into decisions with material consequences for individuals or the organisation
- The system interacts with users who may provide inputs outside the training distribution
- The regulatory environment requires evidence of continuous compliance (EU AI Act, FDA, financial regulators)
- The system was trained or fine-tuned on data that may become stale, biased, or unrepresentative over time
Related Terms
References
- [1] European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Articles 72–73. Official Journal of the European Union, L series.
- [2] Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019) ‘Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations’, Science, 366(6464), pp. 447–453. doi: 10.1126/science.aax2342.
- [3] Parker, W. and Friedman, N. (2021) ‘Zillow to Stop Flipping Homes, Citing Inability to Accurately Predict Prices’, The Wall Street Journal, 2 November.
- [4] U.S. Food and Drug Administration (2024) Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning-Enabled Device Software Functions. FDA Guidance Document.
- [5] Stanford Institute for Human-Centered Artificial Intelligence (2025) Artificial Intelligence Index Report 2025. Stanford University.
- [6] Sato, D., Wider, A. and Windheuser, C. (2019) ‘Continuous Delivery for Machine Learning’, Martin Fowler [blog]. Available at: martinfowler.com/articles/cd4ml.html.
- [7] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J. and Dennison, D. (2015) ‘Hidden Technical Debt in Machine Learning Systems’, Advances in Neural Information Processing Systems (NeurIPS), 28, pp. 2503–2511.
AI Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0
By Jeroen Janssen, Apparens