What is an AI system under the EU AI Act?

Under Article 3(1) of the EU AI Act, an AI system is 'a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.' Critically, the system includes the model, the data pipeline, the human roles, and the organisational decisions — not just the algorithm.

Why is the system the primary unit for AI governance?

Because models do not operate in isolation. A model embedded in a recruitment application with no human oversight poses different risks than the same model embedded in a research tool with expert review. The system boundary defines the governance scope: which data feeds the model, which humans interact with the outputs, which organisational decisions the system influences, and which controls are in place. Governing the model alone misses the majority of the risk surface.

How do you define the boundary of an AI system?

A system boundary is defined by its governance scope: the models it includes, the data pipelines that feed them, the applications that present outputs, the human roles that interact with it, and the organisational decisions it influences. The boundary must be explicit and documented, because the boundary determines which controls apply, which risks are in scope, and who is accountable. An undefined system boundary means undefined accountability.

System — AI Enterprise Control Index Glossary

System

the whole thing. not just the model. the model, the data, the pipeline, the humans, and the organisational decisions that connect them. the thing the regulator asks about when something goes wrong.

"the vendor said 'our model is compliant.' the regulator said 'we don't regulate models. we regulate systems.' the vendor went quiet."

"asked who owns the AI system. got four different answers from four different teams. turns out nobody owns the system. everybody owns a piece."

"our risk assessment covered the model. the EU AI Act requires a risk assessment of the system. those are two very different documents."

Canonical Definition

A bounded socio-technical unit comprising one or more models, applications, data pipelines, and human roles operating under a defined governance scope. The system is the primary unit for risk assessment and conformity assessment under the EU AI Act. The critical distinction: the model is a component of the system; the system includes the model, the data, the interfaces, the human oversight mechanisms, and the organisational decisions that shape how the model’s outputs are used.

Why It Matters

The single most consequential mistake in AI governance is governing the model instead of the system. A model is a mathematical function. A system is that function embedded in an organisational context: fed by specific data, accessed through specific interfaces, overseen (or not) by specific humans, and producing outputs that influence specific decisions. The risk does not live in the model. The risk lives in the system.

The EU AI Act makes this explicit. Article 3(1) defines an AI system, not an AI model. The risk classification (minimal, limited, high, unacceptable) applies to the system. The conformity assessment applies to the system. The documentation requirements apply to the system. An organisation that governs its model but not the system it operates within has governed a component but not the unit that the regulator assesses.

Defining the system boundary is the first governance decision, because the boundary determines everything that follows: which risks are in scope, which controls are required, who is accountable, and what evidence must be produced. An undefined boundary produces undefined accountability. When four teams each own a piece of the system and no one owns the system itself, the regulator finds the gap before the organisation does.

The Stress Test

A regulator asks for the risk assessment of your AI system. You produce a model card documenting the model’s training data, performance metrics, and known limitations. The regulator asks where the system boundary is defined. You point to the model card. The regulator asks about the data pipeline, the human oversight procedures, the escalation mechanisms, the interface design decisions, and the organisational context in which the model’s outputs are used. You have documented none of these.

You governed the model. The regulation governs the system. The gap between those two scopes is the finding.

In the Wild

Recruitment — Amazon, 2018

A Model Problem That Was Actually a System Problem

Amazon built a machine learning system to screen job applicants. The model was trained on ten years of hiring data. The system penalised CVs containing the word “women’s” (as in “women’s chess club”) and downgraded graduates of two all-women’s colleges. The model reflected the biased historical data it was fed. But the failure was not the model — it was the system: the data pipeline that included biased historical data without mitigation, the absence of human review of the model’s recommendations, and the organisational decision to deploy the system without bias testing across protected characteristics.

The model did what it was trained to do. The system had no mechanism to prevent the model from doing harm. Governing the model without governing the system is governing the symptom.

Criminal Justice — COMPAS, 2016

System Boundaries That Nobody Defined

The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) system was used across U.S. courts to assess recidivism risk. ProPublica’s analysis found the system was twice as likely to falsely flag Black defendants as future criminals and twice as likely to falsely label white defendants as low risk. The vendor, Northpointe (now Equivant), argued the model was calibrated. The problem was the system: which data fed it, how scores were presented to judges, whether judges understood the confidence intervals, and whether there was any mechanism for individuals to challenge the scores.

The model was calibrated. The system was not governed. The system boundary included judges, defendants, data, and decisions. The governance boundary included only the algorithm.

Social Services — Dutch Childcare Benefits, 2019

A System That Destroyed 26,000 Families

The Dutch Tax Authority used an automated system to detect fraud in childcare benefit applications. The system flagged applicants with dual nationality at disproportionate rates. The system included the algorithm, the data (which included nationality as a feature), the human review process (which rubber-stamped algorithmic decisions), and the recovery mechanism (which required families to repay benefits immediately, causing bankruptcies and family separations). 26,000 families were affected. The Dutch government fell. The scandal became the defining European case for algorithmic accountability.

Every element of the system failed: the data, the model, the human oversight, and the recovery mechanism. Governing any single element would not have prevented the harm. Only governing the system would.

How to Govern It

Define the system boundary first. Everything else follows from that decision.

Within the AI Enterprise Control Index, system governance is the foundational act:

System Boundary Definition — Explicitly declare what is inside the system: models, data pipelines, interfaces, human roles, and organisational decisions. What is outside the system is also documented. The boundary determines the governance scope.
System Ownership — A single named owner for the system (not for each component). Component ownership exists within layers, but system ownership is the accountability point that regulators and auditors seek.
Risk Assessment at System Level — The EU AI Act requires risk assessment of the system, not the model. The AI Enterprise Control Index provides the structure to ensure that data risks, model risks, interface risks, and human oversight risks are assessed as parts of a single system.
DPIA and FRIA — Data Protection Impact Assessments and Fundamental Rights Impact Assessments operate at the system level, assessing the combined impact of all system components on individuals and groups.
Conformity Assessment — For high-risk systems under the EU AI Act, the conformity assessment evaluates the system as a whole, including its technical documentation, quality management, and post-market monitoring.

When It's Relevant

Always. The system is the unit of governance. Every AI deployment, from a customer-facing chatbot to an autonomous vehicle, is a system. The question is not whether a system exists but whether its boundary, ownership, and controls have been defined. An undefined system is not ungoverned by choice. It is ungovernable by design.

System-level governance becomes critical when:

The AI system is classified as high-risk under the EU AI Act
Multiple teams or vendors contribute components to the same system
The system makes or influences decisions affecting individuals
A regulator or auditor requests documentation of the AI system
Incident investigation requires tracing a failure across multiple components

Related Terms

References

[1] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act). Article 3(1). Official Journal of the European Union.
[2] Dastin, J. (2018) ‘Amazon scraps secret AI recruiting tool that showed bias against women’, Reuters, 10 October.
[3] Angwin, J., Larson, J., Mattu, S. and Kirchner, L. (2016) ‘Machine Bias’, ProPublica, 23 May.
[4] Parliamentary Committee of Inquiry (2020) Ongekend Onrecht (Unprecedented Injustice): Report of the Parliamentary Committee of Inquiry into the Childcare Benefits Affair. Tweede Kamer der Staten-Generaal, The Netherlands.
[5] NIST (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology, U.S. Department of Commerce.
[6] ISO/IEC 42001:2023. Information Technology — Artificial Intelligence — Management System. International Organization for Standardization.
[7] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S. and Vertesi, J. (2019) ‘Fairness and Abstraction in Sociotechnical Systems’, Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (FAT*), pp. 59–68.

AI Enterprise Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0

By Jeroen Janssen, Apparens