the thing the user actually touches. not the model, not the pipeline — the chatbot, the dashboard, the app. the thing that shows up in the screenshot when it goes wrong.
A deployed software system providing user-facing AI functionality. An application may contain one or more agents. The application is the unit of governance; each application requires a System Card. The distinction matters: a model is a statistical artifact, an agent is an autonomous actor, but the application is where capability meets the user and produces real-world effects. You govern applications, not models.
Why It Matters
The AI governance conversation has historically centred on models. Model cards, model evaluation, model risk management. But models do not interact with users. Models do not make promises to customers. Models do not appear in regulatory filings or litigation exhibits. Applications do.
The application is the governance boundary because it is the point where AI capability meets the real world. The same GPT-4 model can power a customer service chatbot (severity 3), a medical triage assistant (severity 5), and an internal code review tool (severity 2). The model is identical. The governance requirements are radically different. That difference is determined by the application, not the model.
This distinction becomes critical in the age of agentic AI. A modern application may contain multiple agents, each calling different models, each with different tool access and autonomy levels. Governing at the model level misses the emergent behaviour that arises from agent orchestration, tool use, and multi-step workflows. The application is the system boundary within which these interactions are contained and controlled.
The Stress Test
An incident occurs: your customer-facing AI assistant incorrectly processes a refund and overcharges a customer’s card. The incident response team asks which system is responsible. The model team says the model performed within specification. The infrastructure team says the API returned the correct response. The agent framework team says the orchestration logic followed its instructions. Nobody owns the application — the integrated system that the customer interacted with.
The investigation stalls because there is no System Card, no designated application owner, no documented architecture showing how the model, agents, tools, and user interface compose into the product that failed. You governed the components. You did not govern the thing the customer used. The customer does not care about your components.
In the Wild
Air Canada’s customer service chatbot — an application combining a language model, a retrieval system, and a conversational interface — fabricated a bereavement discount policy. When challenged, Air Canada argued the chatbot was “a separate legal entity.” The tribunal ruled that the airline was responsible for all information provided through its application, regardless of the underlying technology. The governance failure was not in the model. It was in the application: no System Card, no documented behaviour boundaries, no escalation path when the chatbot hallucinated policy.
The court did not ask about the model. It asked about the application — the thing the customer talked to. The application is the unit of liability because it is the unit of experience.
A major consultancy deployed a multi-agent AI application for financial analysis: one agent retrieved market data, another performed calculations, a third generated narrative summaries, and a fourth validated outputs. Each agent called a different model. The application had model cards for each model but no System Card for the application. When the narrative agent cited a calculation that the validation agent had flagged as uncertain — but which the orchestration layer surfaced without the uncertainty qualifier — the application produced a confident recommendation based on uncertain data. The error was not in any model. It was in the application’s orchestration: the way agents composed their outputs.
Four model cards. Zero application-level governance. The failure mode existed only at the composition layer — the layer that model-level governance does not see.
The EU AI Act defines an “AI system” as “a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.” This definition describes applications, not models. The regulatory architecture is built around the deployed system that interacts with the world — not the statistical artifact inside it.
The regulation already governs at the application level. Organisations that govern at the model level are governing the wrong thing.
How to Govern It
The application is the unit of governance. Every application gets a System Card. No exceptions.
Within the AI Control Index, application governance is centred at L4 (Applications & Agents):
- System Card — Every application has a System Card documenting: purpose, architecture (models, agents, tools, data sources), risk classification, severity level, designated owner, accountable executive, control inventory, and gate history. The System Card is the governance passport for the application.
- Composition Governance — Multi-agent applications require documentation of how agents interact, how outputs are composed, and where information can be lost or distorted in the orchestration layer. The composition is where emergent risks live.
- Gate Enforcement (L4) — Applications pass through lifecycle gates (design, development, deployment, operation) with gate conditions calibrated to the application’s severity level. A severity 4 application has stricter gate conditions than a severity 2 application, regardless of the underlying models.
- HITL Controls — Human-in-the-loop requirements are defined at the application level, not the model level. The same model may require HITL in one application context and not in another.
- Evidence Factory (S1) — Application-level governance artifacts — System Cards, gate records, incident reports, monitoring outputs — are captured and linked to the application in the AI system inventory.
When It's Relevant
Every AI deployment. The application is the governance boundary for all AI systems in the AI Control Index. Whether the application is a customer-facing chatbot, an internal analytics dashboard, or an autonomous agent workflow, it requires a System Card and lifecycle governance at the application level.
Application-level governance is most critical when:
- The application contains multiple agents or calls multiple models — composition risks emerge at the application level
- The application is user-facing — the user interacts with the application, not the model
- Incident response requires understanding the full system, not just individual components
- Regulatory compliance requires documentation of the “AI system” as defined by the EU AI Act
- The same model is used across multiple applications with different risk profiles
Related Terms
References
- [1] European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 3(1). Official Journal of the European Union, L series.
- [2] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D. and Gebru, T. (2019) ‘Model Cards for Model Reporting’, Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*), pp. 220–229.
- [3] Moffatt v. Air Canada (2024) Civil Resolution Tribunal, British Columbia, Canada. Decision No. CRT-2024-00234.
- [4] Anthropic (2025) ‘System Cards: A Framework for AI Application Governance’, Anthropic Research Blog.
- [5] OECD (2024) OECD Framework for the Classification of AI Systems. OECD Digital Economy Papers, No. 349. Paris: OECD Publishing.
- [6] Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Toner, H., Heim, L., Marchal, N., Amodei, D. et al. (2023) ‘Model Evaluation for Extreme Risks’, arXiv preprint, arXiv:2305.15324.
- [7] NIST (2024) Artificial Intelligence Risk Management Framework: Generative AI Profile (AI 600-1). National Institute of Standards and Technology, U.S. Department of Commerce.
AI Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0
By Jeroen Janssen, Apparens