The governance gap in clinical AI
Artificial intelligence in healthcare is no longer experimental. AI systems are reading radiology images, flagging deteriorating patients, recommending treatment protocols, predicting sepsis risk, and triaging emergency presentations. The technology is in clinical use, at scale, today.
Governance has not kept pace. In most health systems, AI tools have been procured through IT or innovation teams without a parallel governance process. There is often no formal risk assessment, no documented clinical validation process, no named accountability owner, and no incident reporting mechanism when AI systems underperform.
The consequences are not theoretical. AI diagnostic tools have been found to perform significantly worse on patients from underrepresented demographic groups. AI triage systems have exhibited age and gender bias. In each case, the technical failure was preceded by a governance failure.
What makes clinical AI high-risk
The EU AI Act's high-risk classification is particularly relevant for healthcare. Any AI system used as a safety component in the provision of healthcare, or intended for use in diagnosis, treatment planning, or patient monitoring, falls within scope. This includes third-party clinical decision support tools purchased from vendors.
If your health system uses an AI tool that influences any clinical decision, a radiologist's read, a triage nurse's escalation, a pharmacist's dosing recommendation, that tool is likely high-risk AI. It requires documented risk management, data governance assessment, technical validation, human oversight mechanisms, and ongoing post-market monitoring.
The four pillars of clinical AI governance
1. Clinical validation and performance documentation
AI systems must be validated in the clinical environment where they will be used, not just in the controlled conditions of the developer's test set. Performance metrics, sensitivity, specificity, false positive and negative rates, must be documented and made available to the clinical teams using the system. Ongoing monitoring is required: clinical AI systems can drift as patient populations and clinical protocols change.
2. Bias assessment and equity monitoring
AI systems trained on historical clinical data inherit the biases in that data. Documented examples include AI systems that assessed skin conditions with significantly lower accuracy for darker skin tones, and sepsis prediction tools that underperformed for elderly patients. Clinical AI governance requires explicit bias testing across relevant demographic groups, before deployment and continuously.
3. Explainability and human oversight
Clinical AI systems must support clinical oversight, not replace it. Clinicians must be able to interrogate AI recommendations and be structurally positioned to override them without institutional pressure to defer. A governance framework that requires documentation for overriding AI, but not for following it, creates implicit automation bias.
4. Incident reporting and learning
When clinical AI systems contribute to adverse events or near-misses, those incidents must be captured and investigated. Clinical AI governance requires an explicit incident reporting mechanism with the same seriousness applied to other patient safety events.
The accountability question
Every AI system in clinical use must have a named accountability owner, a specific clinical leader whose role includes responsibility for that system's performance and the organisation's response when it underperforms. This is not the vendor's responsibility. The deploying organisation is accountable for how AI is used in patient care.
Australian-specific considerations
Australian healthcare organisations face a layered regulatory environment. The TGA regulates software as a medical device, capturing some AI-powered clinical decision support tools. The Privacy Act creates obligations around health information used in AI systems. Organisations with any EU exposure are subject to the EU AI Act's requirements for high-risk healthcare AI.
