when sensitive data leaves your organisation through an AI system because nobody told employees that pasting source code into a chatbot is the same as emailing it to a stranger with a very good memory.
The unauthorised extraction of sensitive data — training data, model weights, or confidential business information — from an AI system. Model exfiltration (extracting the model itself) and data exfiltration (extracting information processed by or embedded in the model) are distinct Security (S2) controls. Exfiltration can be intentional (adversarial extraction attacks) or inadvertent (employees pasting confidential data into external AI services). Both vectors require governance.
Why It Matters
AI systems are data magnets. They ingest, process, and retain information at a scale that makes traditional data loss prevention (DLP) architectures inadequate. The exfiltration surface is no longer limited to email attachments and USB drives. It now includes every prompt window, every API call to an external model, every RAG pipeline that retrieves documents without classification enforcement, and every agent that can make outbound network requests.
The Samsung incident is not an outlier. It is the base case. Surveys consistently show that a significant percentage of enterprise employees paste confidential information into external AI tools. The data enters the provider’s infrastructure. Depending on the provider’s data retention and training policies, it may persist indefinitely. Depending on the model architecture, fragments of that data may be extractable by other users through carefully constructed prompts.
For model exfiltration, the risk is equally structural. Carlini et al. (2021) demonstrated that training data can be extracted from large language models through targeted prompting. This means that any data used in training — or in fine-tuning — is potentially recoverable. If your proprietary data was used to fine-tune a model, that model is now a potential exfiltration vector for your data.
The Stress Test
A client asks you to demonstrate that their confidential data — submitted through your AI-powered advisory tool — cannot be extracted by other users, accessed by your staff outside the engagement team, or retained beyond the contractual period. You check with your engineering team. The data is processed by a third-party LLM API. The API provider’s terms allow data retention for “service improvement.” Your DLP tools do not inspect outbound API payloads to the LLM provider. Your access logs do not distinguish between data processed for Client A and Client B.
You cannot make the assurance the client is asking for. Not because you chose not to — but because the architecture was never designed to provide it. The data boundary does not exist.
In the Wild
In March 2023, Samsung Electronics lifted its internal ban on ChatGPT, allowing employees to use the tool for productivity. Within three weeks, Samsung engineers pasted confidential information into ChatGPT on at least three separate occasions: semiconductor source code, internal meeting notes from a restricted session, and proprietary hardware test sequences. Each submission sent Samsung trade secrets to OpenAI’s infrastructure.
Samsung’s response was immediate and total: a company-wide ban on all external generative AI tools, a 1,024-byte limit on any text submitted to external services, and the initiation of an internal AI development programme to build Samsung-controlled alternatives.
The data left the building through the front door. Employees used the tool exactly as it was designed to be used. The governance failure was not the employees. It was the absence of a classification ceiling.
Researchers at Google, Stanford, Berkeley, and Northeastern demonstrated that GPT-2 could be prompted to emit verbatim memorised training data, including personally identifiable information, copyrighted text, and unique identifiers. The attack required no model access beyond the standard API. The paper identified that larger models memorise more training data, and that data appearing more frequently in the training set is easier to extract.
The implication for enterprise AI: any data used to fine-tune a model, or included in a RAG corpus without access controls, is potentially extractable through adversarial prompting.
The model is the exfiltration vector. If confidential data entered the training pipeline, it can leave through the inference API.
Multiple industry surveys in 2023–2024 found that between 30% and 65% of enterprise knowledge workers use external AI tools without IT approval or oversight. The data submitted includes customer records, financial projections, legal documents, source code, and internal strategy materials. Most organisations have no visibility into what data is being submitted, to which providers, or under what retention terms.
You do not have a ChatGPT policy problem. You have a data classification problem. The AI tool is just the newest exit door in a building with no locks.
How to Govern It
Exfiltration governance is not a policy. It is an architecture.
Within the AI Enterprise Control Index, exfiltration governance spans multiple layers and shields:
- Security (S2) — Implement classification ceilings that enforce the maximum data sensitivity level each AI system can process. Integrate DLP inspection with AI workflows — outbound API calls to LLM providers must be subject to the same content inspection as outbound email.
- Supply Chain (S3) — Evaluate and document the data retention, training, and access policies of every third-party AI provider in your AI SBOM. A model provider that retains prompt data for training is a fundamentally different risk profile than one that processes and discards.
- AI Engineering (L5) — Build data boundaries into the architecture: tenant isolation for multi-client systems, prompt sanitisation pipelines, and output filtering that detects and redacts sensitive data before it reaches the user or downstream systems.
- GRC (S1) — Evidence Factory captures DLP scan results, provider data processing agreements, classification ceiling enforcement logs, and incident records as governance artifacts.
- Data (L6) — Data lineage tracking from ingestion to output. If you cannot trace where data entered the AI pipeline and where it was sent, you cannot govern exfiltration. You can only discover it after the fact.
When It’s Relevant
Every organisation using AI. Every deployment that processes, generates, or transmits data. Exfiltration risk exists whether you are using an external API, a self-hosted model, or an AI agent with tool-calling capabilities. The vectors differ; the governance requirement does not.
Exfiltration risk is highest when:
- Employees have uncontrolled access to external AI services (shadow AI)
- The organisation uses third-party AI APIs without data processing agreements that address retention and training
- AI agents have outbound network access with default-allow permissions
- RAG pipelines retrieve documents without enforcing classification ceilings
- Fine-tuning or training uses proprietary data without evaluating memorisation risk
Related Terms
References
- [1] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A. and Raffel, C. (2021) ‘Extracting Training Data from Large Language Models’, 30th USENIX Security Symposium, pp. 2633–2650. Available at: arxiv.org/abs/2012.07805.
- [2] OWASP Foundation (2025) OWASP Top 10 for Large Language Model Applications v2025.1. LLM02: Sensitive Information Disclosure. Available at: owasp.org/www-project-top-10-for-large-language-model-applications.
- [3] ISO/IEC (2022) ISO/IEC 27001:2022 Information security, cybersecurity and privacy protection — Information security management systems — Requirements. International Organization for Standardization.
- [4] NIST (2024) Artificial Intelligence Risk Management Framework: Generative AI Profile (AI 600-1). National Institute of Standards and Technology, U.S. Department of Commerce.
- [5] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act), Article 15: Accuracy, robustness and cybersecurity. Official Journal of the European Union, L 2024/1689.
- [6] Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A.F., Ippolito, D., Choquette-Choo, C.A., Wallace, E., Tramèr, F. and Lee, K. (2023) ‘Scalable Extraction of Training Data from (Production) Language Models’, arXiv preprint, arXiv:2311.17035. Available at: arxiv.org/abs/2311.17035.
- [7] Raji, I.D., Smart, A., White, R.N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D. and Barnes, P. (2020) ‘Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing’, Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT*), pp. 33–44.
AI Enterprise Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0
By Jeroen Janssen, Apparens