Este artigo está disponível apenas em inglês no momento.
What Is AI Hallucination? The Governance Guide for Enterprise
AI hallucination — when AI models generate confident-sounding but factually wrong content — is not a bug to be fixed. It is a characteristic of how large language models work. How to govern it in enterprise contexts.
Key Takeaways
Hallucination is when an AI language model generates content that is factually incorrect, fabricated, or unsupported by its training data — but presented with the same confidence as accurate content. The model cannot distinguish between what it knows and what it has confabulated.
Hallucination is a fundamental characteristic of current large language models, not a temporary bug. It arises from how these models generate text — predicting statistically plausible next tokens — rather than retrieving verified facts.
The enterprise governance implication: AI-generated content must be verified by humans with appropriate expertise before it is relied upon for consequential decisions. The higher the stakes, the more rigorous the verification required.
High-hallucination-risk use cases that require specific governance: legal document generation (fabricated cases and statutes), medical information (fabricated clinical evidence), financial analysis (fabricated data and research), regulatory submissions (fabricated regulatory references), and journalism (fabricated quotes and events).
Governance measures that reduce hallucination risk: retrieval-augmented generation (constraining AI output to verified source documents), human expert review protocols, citation requirements (requiring AI to cite sources), and domain-specific fine-tuning on verified data.
"Apenas para fins informativos. Este artigo não constitui aconselhamento jurídico, regulatório, financeiro ou profissional. Consulte um especialista qualificado para orientação específica."
AI hallucinations — what they are and why governance matters
An AI hallucination is when an AI system generates output that is confident, plausible-sounding, and factually wrong. The AI doesn't "know" it's wrong — it has no concept of truth. It generates statistically probable text based on patterns in training data, and sometimes those patterns produce fabricated facts, non-existent citations, invented case law, or fictional regulatory references.
Hallucinations are not bugs that will be fixed. They are an inherent property of how large language models work. Models can be made to hallucinate less frequently through better training, retrieval-augmented generation (RAG), and other techniques — but they cannot be eliminated entirely. Any AI governance framework that treats hallucinations as edge cases rather than expected behaviour is inadequate.
Why hallucinations create governance risk
Legal and regulatory. AI-generated legal citations that don't exist (the Mata v Avianca incident in 2023 where a lawyer submitted ChatGPT-fabricated case citations to court); AI-generated regulatory references that are wrong; AI-generated contractual language that doesn't reflect actual terms. The Air Canada chatbot case (Moffatt v Air Canada, 2024 BCCRT 149) demonstrated liability for incorrect AI-provided information.
Clinical. AI-generated medical information that is plausible but wrong. AI radiology tools that identify findings that don't exist (false positives) or miss findings that do (false negatives). Clinical hallucinations can directly affect patient safety.
Financial. AI-generated financial analysis based on fabricated data points. AI credit scoring producing systematically incorrect assessments. AI trading systems acting on hallucinated market signals.
Reputational. Customer-facing AI providing confidently wrong information to customers. AI-generated content published without verification that contains factual errors. The reputational cost compounds when the organisation appears not to have human oversight in place.
Governance controls for hallucinations
Human review for consequential outputs. Any AI output that will be relied upon — in client advice, regulatory submissions, clinical decisions, customer communications, published content — must be reviewed by a qualified human before use. This is not optional. The human reviewer must have the expertise and time to actually evaluate the content, not just rubber-stamp it.
RAG architecture for domain-specific use. Retrieval-augmented generation grounds the AI's responses in verified source documents rather than relying solely on training data. RAG significantly reduces hallucinations for domain-specific questions but doesn't eliminate them. The quality of the source documents and the retrieval mechanism matters.
Confidence thresholds and refusal. Well-configured AI systems can be designed to refuse to answer when confidence is low or the question is outside their reliable domain. For customer-facing AI, escalation to human support when the AI is uncertain is preferable to a confident wrong answer.
Output monitoring. Systematic monitoring of AI outputs for factual accuracy, particularly for high-volume applications (customer service, content generation, data analysis). Sample-based human review with documented methodology.
User awareness. Staff using AI tools must understand that hallucinations are expected, not exceptional. Training should include: never cite an AI-generated reference without verifying it exists; never send AI-drafted client communications without reading them; never rely on AI-generated numbers without checking the source data.
Disclosure and transparency
Where AI outputs are shared with customers, patients, or other external parties, disclosure that AI was involved and that outputs should be independently verified is increasingly a regulatory expectation. EU AI Act Article 50 (effective 2 August 2026) requires disclosure of AI-generated content. California's chatbot law (1 January 2026) requires AI identification. The Australian Privacy Act ADM transparency obligation (10 December 2026) addresses automated decisions.
Primary sources: NIST AI RMF · EU AI Act
Related reading
AI Customer Service Governance · Is My AI Tool Safe? · AI Risk Register Guide