Evidence Factory

the place where “we have governance” becomes “here’s the proof” — or doesn’t.

"we told the auditor we have an evidence repository. it's a sharepoint folder called 'AI Governance' with 200 files and no naming convention. three people have access. one of them left the company."
"asked where our bias test results are stored. got four different answers from four different teams. turns out nobody is wrong because nobody is storing them anywhere specific."
"regulator asked for the artifact chain for our hiring algorithm. we produced it in eleven days. they expected eleven minutes. that was the finding."
Share this one
Canonical Definition

A centralised governance repository that collects, indexes, and retains all evidence artifacts produced by operational layers. Maintained by S1 (GRC). Transforms scattered documentation into an audit-ready evidence chain. The Evidence Factory is not a filing cabinet — it is the infrastructure that makes governance demonstrable.

Why It Matters

Most organisations produce governance evidence. Few can find it. The fundamental problem is not evidence creation — it is evidence fragmentation. Test results live in engineering repositories. Approval records live in email chains. Policy documents live in SharePoint. Incident reports live in ticketing systems. Risk assessments live in spreadsheets. None of these systems talk to each other, and no single query can reconstruct the evidence chain for a given AI system.

When a regulator asks “show me the evidence that this system was tested, approved, and monitored,” the answer should take minutes, not weeks. The Evidence Factory exists to make that possible. It is the architectural commitment that governance artifacts will be collected at point of production, indexed by system and control, version-controlled, and retrievable on demand.

Without an Evidence Factory, every audit becomes a data archaeology project. Teams scramble to reconstruct evidence chains from memory, email archives, and expired links. Documents are found to be outdated, unsigned, or stored in personal drives of employees who have left. The governance existed in practice but cannot be demonstrated in evidence. In regulatory terms, that distinction does not exist.

The Stress Test

Your organisation deploys twelve AI systems across four business units. A supervisory authority sends a request under the EU AI Act: produce the complete conformity evidence for your three high-risk systems within ten business days. The evidence must include technical documentation, risk assessments, data quality evaluations, bias testing results, human oversight procedures, deployment approvals, and post-market monitoring records.

Without an Evidence Factory, each business unit begins a manual search. Engineering checks Git repositories for test results. Legal searches for approved impact assessments. Compliance looks for the risk register. The data team tries to find the data quality report they know they wrote but cannot locate. After eight days, you have assembled approximately 60% of the required evidence, with version conflicts in three documents and no clear audit trail showing when approvals were granted.

With an Evidence Factory, a single query by system identifier returns the complete evidence chain, version-controlled, time-stamped, and linked to the controls they satisfy. The response takes an afternoon, not a crisis.

In the Wild

Financial Services — EBA Guidelines, 2024–2025
The Model Inventory That Could Not Prove Itself

European Banking Authority guidelines on the use of machine learning models require institutions to maintain a model inventory with associated validation evidence. In supervisory examinations across multiple jurisdictions, examiners found that while institutions could list their models, they could not produce the evidence chain linking each model to its validation results, approval records, and ongoing monitoring data in a unified, query-ready format. Evidence existed but was distributed across model risk management tools, shared drives, and email attachments.

The institution had governance. It could not prove governance. The supervisory finding was for the second thing.

Pharmaceutical — FDA 21 CFR Part 11, Ongoing
Electronic Records and the Audit Trail Standard

The pharmaceutical industry’s experience with FDA 21 CFR Part 11 — the regulation governing electronic records and signatures — provides the clearest precedent for what an Evidence Factory must achieve. Since 1997, pharmaceutical companies have been required to maintain electronic records with audit trails, access controls, and version history. Companies that treated this as a documentation exercise failed inspections. Companies that built centralised record management infrastructure passed them.

The AI governance field is now learning the same lesson: evidence management is an infrastructure problem, not a process problem.

Pharma spent two decades learning that audit-ready evidence requires purpose-built infrastructure. AI governance is in year one of the same lesson.

Public Sector — Dutch Algoritmeregister, 2023–2025
Algorithmic Transparency Without Evidence Infrastructure

The Dutch government’s Algoritmeregister requires public sector organisations to register their algorithmic systems with transparency documentation. In practice, many registrations contain policy-level descriptions but lack the underlying evidence: no links to impact assessments, no validation results, no monitoring data. The register demonstrated intent but could not demonstrate operational governance because the evidence infrastructure to feed it did not exist.

A transparency register without an evidence factory behind it is a catalogue of intentions, not a catalogue of controls.

How to Govern It

The Evidence Factory is not a project. It is the infrastructure that makes governance infrastructure auditable.

Within the AI Control Index, the Evidence Factory is a core component of Shield S1 (GRC):

  • Collection at point of production — Evidence is captured when controls execute, not reconstructed after the fact. When a gate is passed, the gate deposits its evidence. When a test runs, the results flow to the factory. This eliminates the gap between “the work was done” and “the evidence was filed.”
  • Indexing by system and control — Every artifact is linked to the AI system it governs and the control it satisfies. This enables per-system evidence retrieval and per-control compliance reporting.
  • Version control and immutability — Evidence artifacts are version-controlled with timestamps and author attribution. Once deposited, artifacts cannot be modified without creating a new version with an audit trail.
  • Retrieval within minutes — The factory supports query-based retrieval: “show me all evidence for System X, Control Y, from the last 12 months.” If the retrieval time exceeds minutes, the factory is not operational.
  • Integration with Mandatory Artifacts — The eight mandatory artifacts (ART-01 through ART-08) are deposited in the Evidence Factory as they are produced, creating the minimum evidence base for each governed AI system.

When It's Relevant

Every organisation governing more than one AI system. The Evidence Factory becomes critical the moment evidence production outpaces a single person’s ability to track, store, and retrieve it manually — which, in practice, means from the second AI system onward.

The Evidence Factory is particularly essential when:

  • The organisation operates high-risk AI systems subject to conformity assessment under the EU AI Act
  • Multiple teams produce evidence across organisational boundaries
  • External audits or regulatory examinations require rapid evidence retrieval
  • The organisation is scaling its AI portfolio and evidence volume is growing faster than manual processes can handle
  • Post-market monitoring generates continuous evidence that must be retained and indexed over the full system lifecycle

See this component in the framework. The Evidence Factory is a core component of Shield S1 (GRC) in the AI Control Index v6.0.

Open Framework →

Related Terms

Evidence Documented proof a control works GRC Governance, Risk & Compliance Mandatory Artifact Required governance documents Gate Mandatory pass/fail checkpoint Control A governed intervention point Post-Market Monitoring Ongoing lifecycle surveillance

References

  1. [1] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act). Official Journal of the European Union, L series. Articles 11 (Technical Documentation), 17 (Quality Management System), 72 (Post-Market Monitoring).
  2. [2] ISO/IEC (2023) ISO/IEC 42001:2023 — Artificial intelligence — Management system. International Organization for Standardization. Clause 7.5 (Documented Information).
  3. [3] FDA (1997) 21 CFR Part 11: Electronic Records; Electronic Signatures. U.S. Food and Drug Administration. Federal Register, Vol. 62, No. 54.
  4. [4] European Banking Authority (2024) Report on machine learning for IRB models. EBA/REP/2024/xx. Section on model inventory and validation evidence.
  5. [5] Raji, I.D., Smart, A., White, R.N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D. and Barnes, P. (2020) ‘Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing’, Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT*), pp. 33–44.
  6. [6] Overheid.nl (2023) Algoritmeregister: Richtlijnen voor registratie van algoritmen door overheidsorganisaties. Ministerie van Binnenlandse Zaken en Koninkrijksrelaties.
  7. [7] NIST (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology, U.S. Department of Commerce. GOVERN 1.1, GOVERN 1.7.

AI Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0

By Jeroen Janssen, Apparens