EU AI Act Governance for Healthcare Organisations
EU AI Act governance for healthcare: classify clinical AI under Article 6, meet deployer duties, and run the FRIA. Annex III deadline: 2 December 2027.
Most hospitals already carry a formidable compliance stack — MDR, IVDR, GDPR, clinical governance, and national licensing regimes. The EU AI Act (Regulation (EU) 2024/1689) does not replace any of that. It adds a layer on top, and the critical skill for healthcare compliance teams is knowing exactly where the new layer attaches and what it demands.
This guide takes the governance-programme angle: how a hospital, health system, or digital-health provider builds the internal processes to classify its AI systems, assign responsibilities, generate evidence, and sustain oversight over time. If you need the detailed analysis of how a specific medical-imaging device gets classified under MDR and the Act, that is covered separately in our medical-imaging AI classification guide. The cross-link from that page back here is intentional — classification is the starting point; governance is the ongoing work.
Two Routes to High-Risk in Healthcare
The Act reaches healthcare AI through two distinct mechanisms under Article 6. Getting the mechanism wrong matters, because each carries a different deadline, a different conformity-assessment route, and different integration requirements.
Route 1 — Article 6(1) + Annex I: AI embedded in medical devices
An AI system is automatically high-risk where it functions as a safety component of a product covered by Union harmonisation legislation listed in Annex I — which includes MDR (EU) 2017/745 and IVDR (EU) 2017/746 — and that product requires third-party conformity assessment by a notified body. Diagnostic imaging AI, AI-assisted surgical planning tools, and IVD interpretation algorithms that carry CE marking under Class IIb or III MDR, or under Class C or D IVDR, fall here. The compliance deadline is 2 August 2028 under the Digital Omnibus agreed in May 2026.
Critically, the EU AI Act requirements for these systems are not assessed separately from the MDR/IVDR certification. They are integrated into the existing notified-body conformity assessment. The medtech provider does not start again from scratch; they extend their technical file, quality management system (Article 17), and risk management documentation to cover the AI Act's specific obligations alongside the MDR/IVDR requirements. The notified body that reviews the device will review AI Act conformity in the same assessment.
Route 2 — Article 6(2) + Annex III: Certain non-device health uses
Annex III, point 5 lists "AI systems intended to be used for dispatching or establishing priority in the dispatching of emergency first-aid services" as high-risk. In practice this covers AI triage tools that determine whether a patient presenting at an emergency department is seen within minutes or hours, and AI dispatch systems used by emergency services. These are not medical devices in the MDR sense — they do not diagnose or treat — but they make consequential access-to-care decisions.
Other Annex III categories can also catch healthcare AI: biometric systems (point 1) used in patient identification or clinical-trial fraud prevention; employment and worker-management AI (point 4) used to monitor clinical staff performance or allocate surgical lists; and Annex III point 5 more broadly where the Act reaches insurance and health-benefit eligibility systems.
Annex III systems face the 2 December 2027 deadline (stand-alone, non-product-embedded AI). They use the standard Article 43 conformity-assessment route — generally internal control under Annex VI for most point 5 systems, though biometric systems under point 1 require the Annex VII notified-body route.
The Article 6(3) filter matters, but use it carefully
An Annex III system is not high-risk if it does not pose a significant risk of harm to health, safety, or fundamental rights — for example, because it performs a narrow procedural task, improves a previously completed human assessment without replacing it, or does only preparatory analytical work. But any system that profiles natural persons is always high-risk, regardless of any other argument. Healthcare providers who want to claim this exemption must document the assessment in writing before deployment and register the system under Article 49(2). "We think it's low-risk" written in a Slack message does not meet the standard.
The Hospital as Deployer: What Article 26 Actually Requires
Most hospitals, health systems, and digital-health operators buying clinical AI from vendors are deployers under Article 26, not providers. The provider is the vendor who developed and placed the system on the market. This matters because deployer obligations are lighter than provider obligations — but they are not trivial, and in healthcare the Article 14 human-oversight duty becomes particularly demanding.
Under Article 26, a deployer must:
- Use the system only for the intended purpose described in the instructions for use, and only under the conditions the provider has specified (Article 26).
- Assign qualified human oversight. Staff who monitor AI outputs must have the competence to understand the system's limitations and to intervene. Ticking a box labelled "clinician reviewed" is not sufficient if the clinician has no training in the system's failure modes (Article 26, Article 14).
- Monitor performance in operation and, where the deployer reasonably suspects the system is not behaving in conformity with the Act, notify the provider and the relevant market-surveillance authority (Article 26).
- Keep logs of operations — Article 26 requires deployers to retain automatically generated logs for at least six months, where those logs are under their control.
- Before deploying AI in a workplace that materially affects clinical or administrative staff, inform workers' representatives (Article 26).
One obligation that catches hospitals by surprise: the Article 27 Fundamental Rights Impact Assessment (FRIA). Article 27 applies to deployers that are public bodies, and to deployers of systems in the creditworthiness or health/life insurance categories of Annex III. Most public hospitals in EU member states are public bodies. Before they put a high-risk AI system into operation, they must conduct a FRIA — a structured assessment of the impact on the fundamental rights of those the system affects, including patient populations. The FRIA must be registered in the EU database under Article 49. In practice this is a multi-week exercise that most hospitals have not yet factored into their procurement timelines.
If a hospital substantially modifies a clinical AI system, or puts it on the market under its own name, Article 25 converts it from a deployer into a provider — inheriting the full provider obligation stack.
Building a Healthcare AI Governance Programme
A governance programme is not a one-time audit. It is the set of processes an organisation runs continuously to maintain visibility, evidence, and control across its AI portfolio. The elements below are grounded in the Act's requirements but designed to be workable without a dedicated AI-compliance department.
Step 1: Build the AI Inventory
You cannot classify or govern what you have not found. Start with a structured inventory of every AI system in use — clinical decision support, scheduling and triage tools, administrative workflow automation, diagnostic aids, staff-monitoring systems, and any AI features embedded in existing clinical software. Include legacy systems in production, pilots in use-limited trials, and systems being evaluated for procurement.
The inventory should record, at minimum: the vendor and product name, the clinical or administrative function, whether patient data is processed, the vendor's classification claim, and the date of last review. A flat spreadsheet works initially. The point is accountability — someone must own each entry.
Step 2: Classify Each System
For each system, work through Article 6. First question: is this an AI system as defined by the Act — an inference-based system, not deterministic rules or pure search? Second: does it fall under Annex I (a product subject to MDR/IVDR requiring notified-body review)? If yes, it is high-risk via Article 6(1), deadline 2 August 2028. Third: does it fall under an Annex III category? If yes, it is presumptively high-risk via Article 6(2), deadline 2 December 2027, unless the Article 6(3) exemption is properly documented. Fourth: does it fall under an unacceptable-risk prohibition in Article 5 — for instance, untargeted scraping of patient facial data, or emotion recognition used to evaluate staff performance? If yes, it must be removed from use, as Article 5 prohibitions have applied since 2 February 2025.
Document each classification decision with a rationale. Regulators will ask for this.
Step 3: Validate Data Quality and Article 10 Representativeness
Vendor technical documentation will not always cover Article 10 requirements for your patient population. Article 10 requires training, validation, and testing data to be relevant, representative, and free from known errors. A diagnostic tool trained predominantly on Northern European cohorts deployed in a Mediterranean health system may carry representativeness gaps that affect performance in practice. Build a clinical validation checklist into procurement: what populations were used for training, how was performance validated across demographic subgroups, and what are the sensitivity and specificity figures under degraded conditions?
This is not just regulatory hygiene. Underperformance in a specific patient subgroup that goes undetected is a patient-safety risk before it is a compliance risk.
Step 4: Design Clinician Oversight under Article 14
Article 14 requires that high-risk AI systems be usable in a way that allows natural persons to understand outputs, question them, and override without friction. For deployers, that obligation lands in clinical workflow design. Document the oversight protocol for each high-risk system: who reviews, with what information, and how overrides are recorded. Include AI oversight in clinical competence frameworks — Article 4 (AI literacy) has applied since 2 February 2025.
A rubber-stamp approval workflow does not satisfy Article 14. The reviewer must have the information needed to make a genuine judgment, not simply confirm what the model said.
Steps 5–6: Monitoring, Incident Reporting, and Documentation
For AI embedded in medical devices (Route 1), post-market monitoring under Article 72 runs alongside MDR Article 83 post-market surveillance. The MDR obligation and the AI Act obligation are not identical — the AI Act specifically requires tracking accuracy drift, model performance against the training distribution, and logging of clinician overrides and their reasons. Use MDR vigilance structures as the backbone and extend them with AI-specific metrics.
When a serious incident occurs, Article 73 places the primary reporting duty on the provider (typically the device manufacturer), with timelines of 15 days from awareness (73(2)), 2 days for critical-infrastructure disruption (73(3)), and 10 days where a death has occurred (73(4)). Deployer hospitals must notify the provider promptly and cooperate with investigation.
Maintain the evidence package for regulators: AI inventory and classification rationale; vendor Article 11 technical documentation; clinical validation records; Article 27 FRIA for public-body deployers; oversight protocols and training records; incident logs; and log retention records (minimum six months under Article 26; ten years for technical documentation under Article 18).
GDPR and Health Data: Article 9 Special Category Processing
Clinical AI almost invariably processes health data — special category personal data under GDPR Article 9. The legal basis in healthcare is typically Article 9(2)(h) (medical treatment by a professional) or Article 9(2)(j) (scientific research). GDPR Article 35 requires a Data Protection Impact Assessment for large-scale health data processing; most clinical AI deployments trigger it.
The EU AI Act does not replace the DPIA. Both apply independently. Run the DPIA and the Article 27 FRIA together — they share evidence and timing — but treat them as distinct obligations for distinct regulatory audiences.
The Penalty Exposure
Non-compliance with the AI Act's high-risk requirements — failure to implement Article 9 risk management, Article 14 human oversight, Article 26 deployer obligations, or Article 27 FRIA — can attract fines under Article 99(4) of up to €15,000,000 or 3% of total worldwide annual turnover, whichever is higher. For a major hospital group or health technology provider with substantial revenues, 3% of worldwide turnover exceeds the fixed sum. Article 5 violations (prohibited practices) carry a higher ceiling of €35,000,000 or 7%. For smaller healthcare operators, Article 99(6) provides that fines are capped at the lower of the percentage or the fixed sum — a proportionality protection, but not an exemption.
The deadlines mean the exposure is real from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I product-embedded AI. The documentation, procurement review, and governance infrastructure required takes substantially longer than most teams expect. Starting the inventory and classification now is not premature.
How Confir Helps Healthcare Organisations
The governance tasks described above generate a substantial compliance workload: classifying dozens of AI systems across two regulatory routes, producing structured evidence for each, running FRIA assessments, and tracking everything in an audit-defensible record.
Confir's rule-based, deterministic classification engine walks compliance and clinical governance teams through Article 5 and Article 6 decisions using plain-English intake questions — same inputs produce the same classification, with the rule that fired shown in human-readable form. It derives the organisation's role (deployer, provider, or both for different systems) and maps each high-risk system to its applicable obligation set, including the Article 27 FRIA workflow for public-body deployers.
The output includes an Article 11 / Annex IV technical documentation pack and an Article 47 / Annex V EU Declaration of Conformity — audit-ready, generated from the same intake data, no consultant required. Pricing starts at €600 per year.
Frequently Asked Questions
Is a hospital an AI provider or a deployer under the EU AI Act?
In most cases, a hospital is a deployer under Article 26 — it uses clinical AI systems developed and placed on the market by vendors. The vendor is the provider. A hospital becomes a provider under Article 25 only if it develops AI in-house and places it on the market, substantially modifies a third-party system, or repurposes it outside the intended use and puts its own name on it. The deployer/provider distinction determines the weight of your obligation stack; hospitals operating as pure deployers have a shorter list but the FRIA and oversight duties are not optional.
Which Annex III categories are most relevant to healthcare?
The primary category for non-device healthcare AI is Annex III, point 5 — which covers AI used to dispatch or establish priority in the dispatch of emergency first-aid services. Biometrics (point 1) catches AI used for patient biometric identification or emotion recognition. Employment management (point 4) applies to AI that monitors, evaluates, or allocates clinical staff. Health and life insurance pricing and risk assessment also falls under point 5. The Annex III categories describe the function of the system, not the sector deploying it — a system that determines patient access to care is point 5 regardless of whether it runs in a hospital or a health app.
Does the EU AI Act apply to medical devices we already have CE-marked?
Yes, but with the later 2 August 2028 deadline for AI embedded in MDR/IVDR products. Existing CE-marked devices containing AI will need to undergo an integrated conformity assessment that covers the EU AI Act requirements alongside MDR/IVDR. Notified bodies are developing their procedures for this. If your current MDR/IVDR certification cycle is due for renewal before August 2028, plan to incorporate AI Act elements into that renewal rather than treating it as a separate exercise.
What does the Article 27 FRIA require for a public hospital?
Article 27 requires a Fundamental Rights Impact Assessment before a public-body deployer puts a high-risk AI system into service. The FRIA must describe the system and its duration of use, identify the categories of persons likely to be affected (including vulnerable groups such as elderly patients, patients with cognitive impairment, or children), assess the likely impact on the fundamental rights of those groups, and describe the measures taken to mitigate any adverse effects. The completed FRIA must be registered in the EU database under Article 49. The FRIA is separate from and does not replace a GDPR DPIA, though both can usefully draw on the same underlying assessment work.
How does the EU AI Act interact with GDPR for clinical AI?
They are independent regulations that both apply. GDPR governs the lawful basis for processing health data (special category data under GDPR Article 9), requires Data Protection Impact Assessments for high-risk processing (GDPR Article 35), and restricts purely automated significant decisions (GDPR Article 22). The EU AI Act imposes risk management, technical documentation, human oversight, and post-market monitoring obligations on high-risk AI systems regardless of data protection status. A clinical AI deployment that processes patient health data will typically need both a GDPR-compliant lawful basis and processing safeguards, and EU AI Act conformity. Running the DPIA and the FRIA concurrently is sensible; treating them as one document is not — they have different legal bases and different regulatory audiences.
What are the deadlines, and does the Digital Omnibus change anything for healthcare AI?
Article 5 prohibited practices applied from 2 February 2025 — any AI system that a healthcare organisation is running that falls into an Article 5 category (untargeted biometric scraping, emotion recognition to evaluate patients against their will, or similar) must already have been removed. For Annex III stand-alone systems (non-device AI including emergency triage), the obligation date under the Digital Omnibus agreed in May 2026 is 2 December 2027. For AI embedded in MDR/IVDR medical devices, the obligation date is 2 August 2028. The Digital Omnibus deferred both deadlines from the original 2 August 2026; use the new dates, not the old ones.
What happens if our vendor's AI system causes patient harm?
The Act allocates regulatory obligations, not civil liability (which stays with national tort law and MDR product-liability regimes). The provider carries primary obligations for technical conformity — risk management, documentation, post-market monitoring. The deployer carries obligations for correct use, oversight, and incident escalation. Where a serious malfunction occurs, the provider reports to market-surveillance authorities under Article 73; the deployer notifies the provider and cooperates. If the deployer substantially deviated from the instructions for use, Article 25 may shift provider-equivalent regulatory liability onto the deployer. Healthcare procurement contracts should allocate these responsibilities and indemnities explicitly before go-live.
Related guides
- Article 6 high-risk medical device classification
- enterprise compliance requirements
- CISO governance framework
- Annex III medical imaging systems
- surgical robot high-risk classification
- AI policy management tools
Manage your EU AI Act compliance in one place
Confir automates risk classification, technical documentation, and audit trails for any company. No consultants. No 6-month projects. 7-day free trial.
Start free trial →