the autocomplete that got so good at predicting the next word that people started asking it for legal advice. and it answered. confidently. incorrectly.
Large Language Model. A general-purpose AI model trained on broad text corpora that generates human-like text, code, or structured outputs. The primary subject of the OWASP LLM Top 10 threat taxonomy. An LLM is a type of model, not a type of system — the distinction matters because governance obligations attach at both the model level (training, evaluation, safety) and the system level (deployment, monitoring, access control).
Why It Matters
LLMs have become the default interface between enterprises and AI. When an organisation deploys “AI,” it increasingly means an LLM: customer service chatbots, internal knowledge assistants, code generation tools, document drafting systems, and decision support applications. The technology is general-purpose, which is both its power and its governance challenge. A single LLM can be used for tasks ranging from benign (summarising meeting notes) to consequential (drafting legal contracts) to dangerous (generating persuasive disinformation) — often within the same deployment.
The OWASP Foundation recognised this unique threat surface by publishing the OWASP Top 10 for Large Language Model Applications, cataloguing ten distinct vulnerability categories specific to LLM deployments. These are not theoretical threats. Prompt injection attacks have compromised production systems. Sensitive data has been extracted through carefully crafted queries. LLMs given tool access have taken actions beyond their intended scope. Each of these incidents occurred in enterprise environments with security teams, code reviews, and deployment gates.
The governance challenge is that LLMs are probabilistic, not deterministic. The same input can produce different outputs. The model cannot explain why it produced a particular response. It cannot distinguish fact from plausible fiction. It will follow instructions that conflict with its intended purpose if those instructions are crafted carefully enough. These are not bugs to be fixed — they are architectural properties of how LLMs work. Governance must account for them as permanent characteristics, not temporary limitations.
The Stress Test
Your organisation deploys an LLM-powered assistant for internal use. Employees use it for drafting emails, summarising documents, generating code, and answering policy questions. The system has access to internal documents through a RAG pipeline. Six months in, you receive a data subject access request asking what personal data the LLM has about a specific individual.
You cannot answer. The LLM does not store data about individuals in an addressable way — but it has been exposed to documents containing personal data through the RAG pipeline, and it may reproduce that data in responses to other users. Your privacy team asks whether the system constitutes “processing” under GDPR. Your legal team says yes. Your engineering team says they have no mechanism to identify what personal data the system has surfaced or to whom. The LLM does not have an access log at the semantic level — it logs queries and responses, but not which fragments of which documents were retrieved and included in which answers.
In the Wild
Samsung semiconductor engineers used ChatGPT to help debug proprietary source code, optimise manufacturing processes, and summarise internal meeting notes. In doing so, they submitted confidential source code, proprietary process data, and internal meeting transcripts to OpenAI’s servers. Three separate incidents were reported within a single month.
Samsung had no policy governing LLM use, no data classification controls preventing confidential data from entering external AI services, and no monitoring system to detect when sensitive information was being submitted. The company subsequently banned employee use of external generative AI tools — a blunt instrument applied because the governance infrastructure did not exist.
The LLM did not steal the data. The employees volunteered it. The governance failure was the absence of classification controls between the user and the model.
The OWASP Foundation published the Top 10 for Large Language Model Applications, establishing the first widely adopted threat taxonomy for LLM-specific vulnerabilities. The ten categories — Prompt Injection, Sensitive Information Disclosure, Supply Chain Vulnerabilities, Data and Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation, and Unbounded Consumption — provided a shared vocabulary for security teams, governance frameworks, and regulatory discussions.
The taxonomy was significant because it demonstrated that LLMs introduce security risks that do not map cleanly onto traditional application security frameworks. Prompt injection, for example, has no equivalent in the OWASP Web Application Top 10. An LLM cannot be secured by the same controls that secure a web application, because the attack surface is fundamentally different: the model itself is the attack surface, not just the application layer around it.
Traditional application security assumes the processing logic is deterministic. LLMs are not. The OWASP LLM Top 10 is the industry’s acknowledgment that a new threat model is required.
Multiple enterprises reported incidents where LLM-powered assistants with tool access performed actions beyond their intended scope. An assistant designed to help with calendar management sent emails on behalf of users without confirmation. A coding assistant with repository access committed code changes that introduced vulnerabilities. A customer service bot with CRM access modified customer records based on misinterpreted instructions.
In each case, the LLM was granted broad tool permissions to be “helpful.” The absence of granular permission boundaries, confirmation requirements for consequential actions, and least-privilege principles meant the LLM interpreted ambiguous instructions in the most action-oriented way possible. The model was doing what it was designed to do: be helpful. The governance failure was in the permission architecture.
An LLM with tools is an agent. An agent with broad permissions and no guardrails is a liability. OWASP LLM06 (Excessive Agency) exists because this lesson keeps being learned the hard way.
How to Govern It
An LLM is a capability, not a solution. Govern the capability, then govern every system that uses it.
Within the AI Control Index, LLM governance spans multiple layers and shields:
- AI Engineering (L5) — The primary control layer. LLM deployment requires evaluation pipelines covering accuracy, safety, bias, and adversarial robustness. Evaluation must be continuous, not one-time. The OWASP LLM Top 10 provides the threat model; the evaluation pipeline provides the evidence that threats are mitigated.
- Security (S2) — Prompt injection defence, input validation, output sanitisation, and data classification enforcement. The model must not receive data above the system’s classification ceiling, and the output must not be executed as code or used to modify systems without validation.
- Observability (S4) — Runtime monitoring of output quality, hallucination rates, prompt injection attempts, and anomalous usage patterns. An LLM in production without observability is a system you trust on faith.
- Applications & Agents (L4) — System-level controls around the LLM: permission boundaries for tool use, human-in-the-loop requirements for consequential actions, output disclaimers, and circuit breakers for automated workflows.
- Data (L6) — Data classification controls preventing sensitive or confidential data from entering LLM contexts without appropriate authorisation and logging.
When It’s Relevant
Every organisation deploying, integrating, or consuming LLM capabilities — whether through direct model access, API integration, or embedded in third-party products. If your employees use ChatGPT, if your applications call Claude, if your products embed Gemini, LLM governance applies.
LLM governance is critical when:
- The LLM has access to internal data through RAG or tool integrations
- The LLM is customer-facing and its outputs create contractual or regulatory exposure
- The LLM has tool access that allows it to take actions (send emails, modify records, execute code)
- Employees use external LLM services with no data classification controls
- The organisation operates in a regulated industry where AI outputs inform consequential decisions
Related Terms
References
- [1] OWASP Foundation (2025) OWASP Top 10 for Large Language Model Applications v2025.1. Available at: owasp.org/www-project-top-10-for-large-language-model-applications.
- [2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017) ‘Attention Is All You Need’, Proceedings of NeurIPS 2017. Available at: arxiv.org/abs/1706.03762.
- [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., et al. (2020) ‘Language Models are Few-Shot Learners’, Proceedings of NeurIPS 2020. Available at: arxiv.org/abs/2005.14165.
- [4] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act), Articles 51–56: Obligations for providers of general-purpose AI models. Official Journal of the European Union.
- [5] NIST (2024) Artificial Intelligence Risk Management Framework: Generative AI Profile (AI 600-1). National Institute of Standards and Technology, U.S. Department of Commerce.
- [6] Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N. and Irving, G. (2022) ‘Red Teaming Language Models with Language Models’, arXiv preprint, arXiv:2202.03286.
- [7] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., et al. (2021) ‘Ethical and social risks of harm from Language Models’, arXiv preprint, arXiv:2112.04359.
AI Control Index v6.0 · Glossary · June 2026 · i-DEPOT 158508 (BOIP) · CC BY-NC-ND 4.0
By Jeroen Janssen, Apparens