AI Hallucination Risk Under the EU AI Act

Guide28 January 2026· 10 min read

AI hallucination under the EU AI Act: Articles 9, 13, 14 and 15, the 2 December 2027 deadline, and mitigations including RAG and human oversight design.

A legal-research tool that invents case citations. A medical summariser that fabricates a drug dosage. Both are examples of the same underlying phenomenon: a model producing output that sounds authoritative and is wrong. Regulators don't need a technical name for it. The EU AI Act simply asks whether you identified the risk, assessed its severity, and put controls in place. If your system is high-risk and you haven't, the fine ceiling is €15 million or 3% of worldwide turnover — whichever is higher.

What Hallucination Actually Means

Hallucination is a model generating output that is confident but false or unfounded — invented case law, fabricated dosage figures, financial history that doesn't exist in the source documents. It isn't a bug in the traditional sense. It emerges from how probabilistic language models work: the model predicts the next plausible token, not the next true one.

The EU AI Act, Regulation (EU) 2024/1689, does not use the word "hallucination." It does not need to. Article 9(1) requires providers of high-risk AI systems to establish, implement, and maintain a risk management system that identifies, analyses, and mitigates "known and reasonably foreseeable risks" to health, safety, and fundamental rights. Output-accuracy failure in a high-stakes decision context is precisely that kind of risk.

Which Systems Must Manage This Risk

Hallucination becomes a compliance obligation only when your system lands in the high-risk tier. That means either Article 6(1) — a safety component of a product regulated under Annex I, such as a medical device — or Article 6(2) — a system falling into one of the eight Annex III use-case areas.

The Annex III categories most exposed to hallucination risk are:

Employment and worker management (Annex III, point 4) — a recruitment tool that invents candidate qualifications or fabricates employment history.
Access to essential services (Annex III, point 5(b)) — a credit-assessment system that generates financial data not present in the applicant's documents.
Education and vocational training (Annex III, point 3) — an exam-evaluation system that misattributes answers or invents a student's prior record.
Law enforcement (Annex III, point 6) — a case-research tool that fabricates precedent, skewing a risk assessment.

If your system falls into one of these categories, hallucination is not a product-quality issue to handle internally. It is a named compliance risk that must appear in your risk management documentation.

The Article 6(3) Filter

Under Article 6(3), a system that technically falls into an Annex III category is not high-risk if it poses no significant risk of harm to health, safety, or fundamental rights — for example, because it performs a narrow procedural task or does only preparatory work without influencing a human assessment. Hallucination undermines any such claim. A system that routinely generates false outputs cannot credibly be framed as low-harm. Any provider attempting to self-certify out of high-risk status must document the analysis — and that documentation will not hold up if the system's known failure modes include fabricated information used in consequential decisions.

The Four Articles That Matter

Article 9 — Risk Management

Article 9(1) requires a risk management system that runs throughout the system's lifecycle. Article 9(2) extends the scope to both intended use and "reasonably foreseeable misuse." For a document-synthesis tool, the intended-use risk is the model hallucinating from incomplete data; the misuse risk is a user relying on the output without the human review step you specified.

Article 9(3) requires written documentation of the risk identification and analysis process. Noting "model may produce inaccurate outputs" in a one-line risk register is not sufficient. You need to trace the failure mode to the concrete harm: which decisions are affected, who is harmed, how severely, and how often.

Article 15 — Accuracy, Robustness, Cybersecurity

Article 15(1) requires that high-risk AI systems achieve an appropriate level of accuracy, and that the declared accuracy metrics be specified in the instructions for use. The practical implication: you cannot leave accuracy as an undefined ambition. You must test, measure, and disclose. If your system produces hallucinated outputs in a measurable share of cases, that rate must appear in the technical documentation — and your risk management controls must bring it to an acceptable level.

"Appropriate" level of accuracy is context-dependent. A medical summariser used to support clinical decisions is held to a much higher standard than a general-purpose content tool. The severity of potential harm sets the bar.

Article 13 — Transparency to Deployers

Article 13 requires providers to supply deployers with clear, accurate information about the system's capabilities and limitations. For any system with known hallucination tendencies, this means the instructions for use must explicitly state: what the system can and cannot reliably do, which output types are most prone to error, and what human oversight steps the deployer must put in place before acting on the output.

This isn't optional language in a terms-of-service footer. Regulators and deployers will look to it as the document that defined what the deployer was supposed to do.

Article 14 — Human Oversight

Article 14 requires that high-risk AI systems be designed so that deployers can effectively oversee them during use. For hallucination-prone systems, that design obligation is concrete: the interface must not present outputs as definitive facts, the system must indicate when it is drawing inferences rather than citing source text, and human review must be structurally required before any output triggers a high-stakes decision.

Article 14 is the backstop. If your technical mitigations reduce but don't eliminate hallucination risk, mandatory human review is the final layer — and it must be built into the workflow, not left as a suggestion.

GPAI Models and LLMs: A Note

GPAI model providers must document hallucination as a known limitation under Article 53 — specifically in the technical documentation supplied to downstream providers, which must cover accuracy boundaries and known failure modes. For providers designated as systemic-risk under Article 51 (the presumption applies above 10²⁵ FLOPs), Article 55 adds adversarial-testing obligations directly relevant to hallucination at scale.

GPAI obligations under Chapter V have applied since 2 August 2025. The Digital Omnibus deferral that moved the Annex III high-risk deadline to 2 December 2027 does not affect Chapter V.

Practical Mitigations

The Act does not prescribe specific technical methods. It requires that whatever measures you implement are proportionate to the identified risk and demonstrably effective. In practice, the controls that map most clearly to Article 9 and Article 15 are:

Retrieval grounding (RAG): Constrain the model to generate output only from retrieved source documents, with explicit citations. A medical summariser that cites the specific paragraph it is drawing from cannot fabricate a dosage that doesn't appear there — or, if it does, the citation exposes the error immediately.

Confidence thresholds: For each factual claim, assign a confidence score and flag anything below a defined threshold for human review. Document the threshold-setting rationale in the risk management file.

Constrained output formats: Where the output is structured — a credit narrative, a candidate assessment — define the schema and validate against it post-generation. A field that requires a date cannot be populated with a plausible-sounding but invented figure if validation rejects non-date inputs.

Evaluation against benchmarks: Test the system systematically before deployment and after any significant change. Record hallucination rates by use case and use them to justify the "appropriate level of accuracy" claim in the Article 15 disclosure.

Disclosure to users (Article 50 where applicable): Article 50 applies to certain AI systems interacting with natural persons — chatbots, emotion recognition, synthetic-content generators. If your system is customer-facing and falls within Article 50's scope, disclosure of AI involvement is required from 2 August 2026 regardless of whether the system is also high-risk. That disclosure should address limitations honestly, including output-accuracy bounds.

How Confir Helps

Confir's assessment covers accuracy and oversight controls as part of the structured compliance workflow. The AITR area (Data & Technical Robustness, covering Article 15) captures accuracy specifications and mitigation measures; the AITO area (Transparency & Human Oversight, covering Articles 13 and 14) captures the deployer-instruction requirements and oversight design. Both are part of the same assessment run — you document the risk once, and the output feeds directly into the Article 11 / Annex IV technical documentation pack.

The engine is rule-based and deterministic. It does not generate findings probabilistically. That design choice matters for a compliance product: same intake, same finding, the rule that fired is human-readable.

Frequently Asked Questions

Is AI hallucination a defined term in the EU AI Act?

No. The Act uses the language of "known and reasonably foreseeable risks" (Article 9) and "appropriate level of accuracy" (Article 15). Hallucination — confident but false or unfounded model output — fits squarely within both framings. The absence of the word from the text does not reduce the obligation; it means you are responsible for translating the technical phenomenon into the Act's risk-management vocabulary.

Does hallucination risk apply only to high-risk AI systems?

The binding obligations — Article 9 risk management, Article 15 accuracy specification, Article 13 disclosure of limitations, Article 14 oversight design — apply only to high-risk systems. For limited-risk systems subject to Article 50 (customer-facing chatbots, for example), there is a disclosure obligation but no mandatory risk management system. Minimal-risk systems have no statutory accuracy obligations. That said, hallucination causing real harm can create liability outside the AI Act entirely — consumer protection, professional negligence, GDPR Article 22 — so the commercial case for managing it exists regardless of tier.

What does Article 9 actually require me to document?

Article 9(3) requires a written risk identification and analysis — not a generic checklist but a traced causal chain: the specific failure mode, the downstream decision it affects, the population that can be harmed, severity and likelihood estimates, and the basis for those estimates (testing data, expert consultation, field evidence). Article 9(5) requires that the mitigations you implement address what you identified. The risk management file has to hold together as a coherent document a regulator can follow.

How does Article 15 interact with accuracy in practice?

Article 15(1) requires that accuracy levels be specified in the instructions for use. Before you can write that specification, you need to have measured accuracy — which means structured testing against defined benchmarks before deployment. Post-deployment monitoring under Article 72 then feeds back into whether the declared accuracy levels remain valid. For a system with hallucination risk, you are committing to a rate (implicitly or explicitly) and demonstrating you have controls to stay within it.

What is the compliance deadline for high-risk systems?

Under the Digital Omnibus agreed in May 2026, stand-alone high-risk AI systems listed in Annex III must comply from 2 December 2027 (pushed back from the original 2 August 2026 date). High-risk AI embedded in regulated products covered by Annex I must comply from 2 August 2028. GPAI model obligations under Chapter V have applied since 2 August 2025 and were not deferred.

Does a deployer have separate obligations on hallucination?

Yes. Article 14 requires deployers to implement human oversight that can detect and correct problematic outputs before they trigger a decision. Article 13 requires providers to supply the instructions that tell deployers how to do that — so if a deployer skips the oversight step, the question of whether they were adequately instructed becomes material. Deployers are also expected to monitor the system's performance in their specific context (Article 26) and flag accuracy issues back to the provider.

Can retrieval-augmented generation (RAG) satisfy the Article 9 risk management requirement?

RAG is a strong mitigation and directly addresses the most common hallucination failure mode by grounding outputs in verified source documents. It does not, on its own, satisfy Article 9. The Article requires a documented system: risk identification, analysis, mitigation measures, testing results, and post-market monitoring. RAG is one element of the mitigation layer. You still need the documentation, the accuracy testing, the human oversight design, and the ongoing monitoring — all traceable to the risk you identified.

Manage your EU AI Act compliance in one place

Confir automates risk classification, technical documentation, and audit trails for any company. No consultants. No 6-month projects. 7-day free trial.

Start free trial →