Hallucination

Why It Matters

Hallucination is one of the few AI risks that gets more dangerous as the model gets better. A mediocre model produces visibly clumsy output. A frontier model produces hallucinations that are linguistically indistinguishable from fact. The better your AI writes, the harder its fabrications are to catch.

This is a board-level problem, not a prompt-engineering problem. When AI-generated content enters legal filings, regulatory submissions, board papers, customer communications, or medical records, a hallucination is no longer an amusing chatbot glitch. It is a governance failure with documented consequences: sanctions, liability, reputational destruction, and regulatory intervention.

The structural challenge is that hallucination cannot be eliminated. It can be reduced through retrieval-augmented generation, fine-tuning, evaluation pipelines, and output grounding. But the residual rate is never zero. Governance, therefore, must assume hallucinations will reach production and design the detection, escalation, and correction mechanisms accordingly.

The Stress Test

A regulator asks your organisation to demonstrate that outputs from your customer-facing AI are factually accurate. You point to your RAG pipeline. The regulator asks for the evaluation results. You have none — because the pipeline was never tested against ground truth after deployment. Your monitoring tracks latency, throughput, and error rates. It does not track whether the answers are true.

The system has been live for nine months. You cannot say how many hallucinated responses reached customers. You cannot say because you have no mechanism to count them. That absence is the finding.

In the Wild

How to Govern It

You cannot eliminate hallucination. You can govern the blast radius.

Within the AI Enterprise Control Index, hallucination governance spans multiple layers and shields:

• Observability (S4) — Runtime detection of output quality degradation, confidence monitoring, and factual grounding checks. If you are not measuring whether outputs are true, you are not monitoring. You are logging.
• AI Engineering (L5) — Retrieval-augmented generation pipelines with citation verification, evaluation plans with ground-truth benchmarks, and model evaluation gates that test for factual accuracy before deployment.
• Ethics & Fairness (L2) — Explainability standards requiring that AI outputs include provenance indicators. A user should know whether a response is grounded in retrieved evidence or generated from the model’s parametric memory.
• GRC (S1) — Evidence Factory captures hallucination detection test results, evaluation benchmarks, and incident records as governance artifacts.
• Applications & Agents (L4) — System-level controls: output disclaimers, confidence thresholds below which responses are withheld or escalated, and circuit breakers for automated workflows.

When It's Relevant

Every deployment of a generative AI system. Without exception. Hallucination is not a risk that applies only to high-risk systems. A customer-facing chatbot that fabricates a return policy (Air Canada), a legal research tool that invents case law (Mata v. Avianca), and an internal assistant that hallucinates compliance status are all hallucination incidents — each with different blast radii, all requiring detection.

Hallucination risk is highest when:

• The query falls outside the model’s training distribution
• The system has no retrieval layer to ground responses in verified documents
• The output is consumed by humans who lack the domain expertise to detect fabrication
• The output enters an automated workflow with no human review checkpoint
• The organisation has no post-deployment evaluation measuring factual accuracy

Related Terms