AI Governance for US Healthcare Organisations: FDA, HIPAA, CMS, and State Requirements

FDA regulation of clinical AI

The FDA regulates Software as a Medical Device (SaMD) through its Center for Devices and Radiological Health. AI intended for diagnosis, cure, mitigation, treatment, or prevention of disease is generally regulated as a medical device. Most AI cleared through the 510(k) pathway or De Novo pathway for diagnostic AI in radiology, ophthalmology, cardiology, and pathology. Healthcare organisations must verify that clinical AI has appropriate FDA clearance before deployment — using uncleared AI in clinical settings creates significant legal exposure. Check the FDA's 510(k) and De Novo databases at accessdata.fda.gov.

HIPAA and AI

HIPAA applies to all AI tools that create, receive, maintain, or transmit protected health information (PHI). The most common governance failure: using AI tools without a Business Associate Agreement. Most general-purpose AI tools are not HIPAA compliant without specific BAA and configuration. Verify that every AI tool touching PHI has a signed BAA and meets HIPAA Security Rule requirements — encryption, access controls, audit logging, and breach notification procedures.

CMS and Medicare Advantage AI

CMS guidance (2024) clarified that Medicare Advantage plans must ensure utilisation management decisions informed by AI are based on individual patient circumstances rather than population-level statistical averages. This was a direct response to Senate investigations finding some MA plans used AI to systematically deny claims. Plans with unusually high AI-assisted denial rates should expect heightened CMS scrutiny. Claims review must involve a qualified clinician who genuinely considers the individual patient's specific clinical situation.

Clinical responsibility

GMC (and equivalent state medical boards) guidance is consistent: clinicians retain professional responsibility for decisions made with AI assistance. Before deploying AI in a clinical pathway, ensure: clinicians have received adequate training on the tool's capabilities, limitations, and failure modes; a clinical lead is designated for oversight; ongoing performance monitoring is in place; and there is a clear process for reporting AI-related clinical concerns and for reviewing AI performance changes.