AI governance in healthcare.
In every other sector, AI governance is about managing risk to the organisation. In healthcare, AI governance is about patient safety. That single distinction reshapes every governance decision.
The regulatory landscape
Healthcare AI sits at the intersection of three regulatory regimes simultaneously, and the interactions between them are not always obvious. Getting the analysis wrong creates compliance gaps that may not surface until a regulator inquiry or an adverse event.
- EU AI Act: AI used in healthcare contexts is frequently classified as high-risk, particularly clinical decision support, diagnostic AI, and AI used in patient triage. The Act applies cumulatively to AI systems that are also medical devices.
- Medical Device Regulation (MDR) and IVDR: AI systems that meet the definition of a medical device require CE marking and conformity assessment under MDR or IVDR, depending on their intended use. The MDCG 2019-11 guidance provides interpretation for software as a medical device.
- FDA pathway (US): AI/ML-enabled medical devices follow FDA's regulatory pathway with the FDA's evolving framework for AI/ML-based software as a medical device (SaMD). The predetermined change control plan introduced in 2023 has reshaped how iterative AI medical devices are regulated.
- HIPAA and equivalent privacy regimes: patient health information used to train or operate AI systems is subject to stringent privacy obligations that apply on top of general data protection law.
- Clinical governance and professional regulation: the doctor, nurse, or allied health professional using AI in clinical decisions retains professional responsibility. AI does not transfer clinical accountability.
Where governance most often fails
Four governance failures recur across healthcare AI deployments we have examined. Each is preventable; each has been associated with documented adverse outcomes.
- Performance drift in clinical AI without monitoring: models that performed well in validation produce different outcomes once deployed because the patient population, presentation patterns, or clinical workflow has changed. Without active monitoring, this drift goes undetected.
- Inadequate validation across demographic subgroups: AI systems validated on aggregate populations may perform substantially worse for minority subgroups. In healthcare contexts, this is not just a fairness issue; it is a patient safety issue.
- Automation bias in clinical decision support: clinicians under time pressure increasingly defer to AI recommendations rather than independently evaluating them. The AI was designed to support clinical judgment; in practice, it substitutes for it.
- Procurement gaps with AI medical device vendors: healthcare organisations procuring AI medical devices often lack the technical and regulatory expertise to assess vendor representations, conformity assessment status, and post-market surveillance commitments.
Healthcare AI guidance
AI Governance in Healthcare: Patient Safety as the Governance Framework
ReadModel Risk Management for Clinical AI Systems
ReadGDPR and the EU AI Act: Health Data Considerations
ReadAI Vendor Due Diligence for Healthcare Procurement
ReadAI Incident Response in Clinical Settings
ReadHow to Audit Clinical AI Systems
ReadKey governance questions for healthcare AI leaders
These are the questions regulators, accreditation bodies, and board risk committees are increasingly asking healthcare organisations about their AI use.
Can you produce a complete inventory of every AI system in clinical and administrative use, with its risk classification under the EU AI Act?
For AI systems classified as medical devices, do you have current CE marking or FDA clearance documentation from your vendors?
What is your process for detecting and responding to AI model performance degradation in clinical settings?
Have your AI clinical decision support tools been validated on patient populations representative of your actual patient demographics?
What is your policy on clinician reliance on AI recommendations, and how is that policy enforced?
Do your AI procurement contracts include post-market surveillance commitments, incident notification obligations, and audit rights?
Have you conducted a fundamental rights impact assessment for AI systems used in patient triage or resource allocation?
Free assessment
How mature is your healthcare AI governance?
Our six-question AI governance assessment takes five minutes and identifies your maturity level across visibility, accountability, policy, oversight, regulatory exposure, and board engagement.
Take the free assessment