when someone asks “where did the model get that data?” and nobody can answer. which is most of the time.
The documented chain from data source through transformation, storage, and consumption by an AI system. Required for audit defensibility, bias investigation, and GDPR data subject access requests. Without lineage, a model is a black box fed by a black pipeline connected to black sources — and every question about its behaviour becomes unanswerable.
Why It Matters
Data lineage is the foundation on which every other AI governance control depends. You cannot audit what you cannot trace. You cannot investigate bias if you do not know which datasets entered the pipeline. You cannot fulfil a GDPR Article 15 data subject access request if you cannot determine whether an individual’s data was used in training. You cannot prove lawful basis for processing if you cannot identify the data’s origin.
The challenge is structural. Modern AI systems consume data from dozens of sources, each with different consent bases, retention periods, and classification levels. Data is joined, transformed, augmented, deduplicated, and sampled before it ever reaches a model. Each step is an opportunity for governance metadata to be lost. Most organisations track the data they store but not the data they transform — and it is the transformations that determine what the model actually learned.
Regulatory pressure is accelerating. The EU AI Act (Art. 10) requires high-risk AI providers to implement data governance practices covering provenance, collection processes, and preparation operations. The GDPR’s right to explanation (Recital 71) is effectively unenforceable without lineage. The NIST AI RMF maps data provenance to Govern 1.5 and Map 2.3. None of these frameworks accept “we don’t know where the data came from” as an answer.
The Stress Test
A data subject exercises their GDPR Article 15 right and asks whether their personal data was used to train your customer scoring model. Your data team checks the model registry. The model card lists a training dataset name but not its sources. The dataset was assembled eighteen months ago by a contractor who has since left. The shared drive contains a CSV with no metadata, no schema documentation, and no record of consent basis.
You have thirty days to respond. You cannot confirm or deny whether the individual’s data is in the training set because you have no lineage connecting the original data collection to the final training artifact. The model is live. It scores 200,000 customers per month. You have no lineage because nobody asked for it when the project launched. That absence is now a compliance incident.
In the Wild
Clearview AI scraped over three billion facial images from the public internet to train its facial recognition system. When regulators in France (CNIL), Italy (Garante), the UK (ICO), and Greece (DPA) investigated, they found no documented lineage connecting individual images to lawful processing bases. The company could not demonstrate consent, could not identify which images belonged to which data subjects, and could not honour deletion requests because there was no mapping from source data to trained model parameters.
Combined fines exceeded €60 million across jurisdictions. The technical violation was straightforward: the data existed in the model, but the lineage connecting person to data to model to output did not.
You cannot delete what you cannot trace. You cannot trace what you did not document. The fine is for the documentation you never created.
A widely used healthcare algorithm by Optum was found to systematically discriminate against Black patients. The model used healthcare costs as a proxy for healthcare needs. Because Black patients historically had less access to care (and therefore lower costs), the model learned that Black patients were “healthier” than equally sick white patients. The bias was not in the model architecture — it was in a data transformation decision made early in the pipeline that equated spending with need.
Researchers at UC Berkeley estimated the algorithm affected 200 million patients annually. Tracing the bias required reverse-engineering the feature pipeline because no lineage documentation existed connecting the raw claims data to the proxy variable to the model’s scoring logic.
The bias was introduced in a data transformation step. Without lineage, the investigation took a published academic paper to find what documentation should have made obvious.
ImageNet — the dataset that underpins much of modern computer vision — was found to contain non-consensual intimate images, racist labels, and personally identifiable photos scraped from Flickr without meaningful consent. Researchers from Excavating AI documented that the dataset’s “person” categories included labels like “alcoholic” and “kleptomaniac” applied to real, identifiable people.
Thousands of models were trained on ImageNet. None of them had lineage documentation connecting their weights back to these specific problematic images and labels. When the dataset was partially cleaned, there was no mechanism to identify which downstream models were affected.
The internet’s most influential dataset had no lineage. Every model trained on it inherited the problem. Most still do.
How to Govern It
Lineage is not a data engineering project. It is a governance infrastructure requirement.
Within the AI Control Index, data lineage governance spans multiple layers and shields:
- Data (L6) — The primary control layer. Every dataset entering the AI pipeline must be catalogued with source metadata, consent basis, classification level, and retention policy. Every transformation must be versioned and reproducible. The lineage chain must be unbroken from source to model.
- GRC (S1) — Evidence Factory captures lineage artifacts as governance evidence: data catalogues, transformation logs, consent records, and provenance documentation. These are the artifacts a regulator will request.
- Supply Chain (S3) — Third-party data and pre-trained models require lineage documentation from the supplier. If your vendor cannot provide lineage, you are inheriting their governance debt — and their liability.
- AI Engineering (L5) — The model registry must link every model version to its exact training data snapshot. Evaluation results must trace back to the data used for evaluation. Without this link, model cards are documentation theatre.
- Ethics & Fairness (L2) — Bias investigation requires lineage to trace discriminatory outcomes back to their data source. Without lineage, fairness audits are guesswork.
When It’s Relevant
Every AI system that consumes data — which is every AI system. Lineage is not a high-risk-only requirement. Any model that processes personal data triggers GDPR obligations that are unenforceable without lineage. Any model that makes consequential decisions triggers fairness obligations that are uninvestigable without lineage.
Data lineage becomes critical when:
- A data subject exercises access, deletion, or rectification rights under GDPR
- A bias investigation requires tracing discriminatory outputs to their data source
- An auditor requests evidence of lawful data processing for AI training
- A model is retrained and you need to verify the new dataset has not introduced prohibited data
- A third-party dataset or pre-trained model enters your pipeline and you must verify its provenance
Related Terms
References
- [1] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act), Article 10: Data and data governance. Official Journal of the European Union.
- [2] Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019) ‘Dissecting racial bias in an algorithm used to manage the health of populations’, Science, 366(6464), pp. 447–453. doi: 10.1126/science.aax2342.
- [3] Commission Nationale de l’Informatique et des Libertés (CNIL) (2022) Facial recognition: the CNIL orders Clearview AI to stop processing data. Decision No. MED-2022-014.
- [4] Crawford, K. and Paglen, T. (2021) ‘Excavating AI: The Politics of Training Sets for Machine Learning’, Excavating AI. Available at: excavating.ai.
- [5] NIST (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology, U.S. Department of Commerce. Govern 1.5, Map 2.3.
- [6] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Daumé III, H. and Crawford, K. (2021) ‘Datasheets for Datasets’, Communications of the ACM, 64(12), pp. 86–92. doi: 10.1145/3458723.
- [7] European Data Protection Board (2024) Guidelines on the use of personal data in AI model development and deployment. EDPB Guidelines 2024/01.
AI Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0
By Jeroen Janssen, Apparens