AI Governance for US Healthcare Organisations: FDA, HIPAA, CMS, and State Requirements

Healthcare AI in the US is regulated by FDA as a medical device, subject to HIPAA for data handling, and faces increasing CMS oversight for AI in Medicare and Medicaid decision-making. Here is the governance framework.

Key Takeaways

AI used for diagnosis, treatment recommendation, risk prediction, or patient monitoring is regulated by FDA as Software as a Medical Device (SaMD). FDA clearance (510(k)), De Novo authorisation, or PMA approval is required before clinical deployment.
HIPAA's Privacy and Security Rules apply to all AI systems that create, receive, maintain, or transmit protected health information. Business Associate Agreements must be in place with AI vendors before any PHI is processed.
CMS issued guidance in 2024 requiring Medicare Advantage plans to ensure AI-driven prior authorisation decisions are based on individual patient circumstances — not population-level statistical models. Plans whose AI systematically overrides individual clinical presentations face enforcement.
ONC's HTI-1 rule includes transparency requirements for AI algorithms in certified health IT — algorithms must be disclosed to clinicians who use AI-assisted decision support tools.
Clinical validation — demonstrating AI tool performance in your specific patient population — is a regulatory and liability requirement. Vendor validation studies on different populations may be insufficient.
Healthcare organisations face clinical negligence liability for AI-assisted decisions that cause patient harm. Clinical staff must be trained on AI tools' limitations — reliance on AI without understanding its capabilities does not satisfy the professional standard of care.

"情報提供のみを目的としています。この記事は法律、規制、財務または専門的なアドバイスを構成するものではありません。具体的なアドバイスについては、資格を持つ専門家にご相談ください。"

US healthcare AI governance — fragmented federal/state landscape

The US healthcare AI regulatory environment in 2026 is the most fragmented of any major jurisdiction. There is no single federal AI healthcare law; instead, requirements emerge from multiple federal agencies (FDA, HHS, CMS, ONC, OCR, FTC), 50 different state legislatures, and parallel professional standards. The result is a complex compliance puzzle that varies depending on whether the AI is a regulated medical device, processes protected health information, makes clinical decisions, or interacts with patients directly.

Two structural shifts shape the 2026 landscape. First, the December 2025 Executive Order 14365 ("Ensuring a National Policy Framework for Artificial Intelligence") signalled federal policy to challenge state AI rules viewed as overly burdensome — though as of mid-2026 this has not translated into preemption legislation. Second, in the absence of comprehensive federal AI legislation, states have moved aggressively: in 2025, 47 states introduced more than 250 bills including health AI regulation, with 33 becoming law in 21 states (per Manatt Health's AI policy tracker). 2026 has continued the trend with approximately 200 state AI bills tracked through Q1.

FDA — AI as medical device

The Food and Drug Administration is the primary federal regulator for AI as a medical device. As of July 2025, the FDA had authorised more than 1,250 AI-enabled medical devices. The FDA's oversight is grounded in premarket review and a risk-based approach requiring devices to demonstrate "a reasonable assurance of safety and effectiveness."

The FDA's January 2026 guidance on clinical decision support software clarified that AI products directly influencing clinical judgment or patient management will likely require FDA review or clearance. AI that meets the regulatory definition of a medical device requires: classification (Class I, II, or III based on risk); premarket notification (510(k)) or premarket approval (PMA) depending on classification; quality system regulation compliance; post-market surveillance; and adverse event reporting through MedWatch. Software as a Medical Device (SaMD) and AI/ML-enabled SaMD have specific FDA frameworks including the Predetermined Change Control Plan (PCCP) approach for managing post-market AI updates.

HIPAA and AI — privacy and security

The Health Insurance Portability and Accountability Act (HIPAA) governs use of protected health information (PHI). When AI processes PHI for clinical, operational, or analytics purposes — whether in training, deployment, or integration — HIPAA compliance is required. Key obligations: covered entities must execute Business Associate Agreements (BAAs) with AI vendors before sharing PHI; data must be encrypted in transit and at rest; access controls must limit PHI to authorised personnel only; security incident response plans must address AI-specific scenarios; and breach notification requirements apply if AI systems expose patient data. The Office for Civil Rights (OCR) enforces HIPAA, with penalties up to $1.5 million per violation category per year.

The HHS strategy for AI in Health, Human Services, and Public Health (issued January 2025, with the comprehensive HHS AI strategy released 4 December 2025) provides framework guidance on AI use in HHS-regulated activities. The 4 December 2025 21-page HHS strategy directs internal AI use across the Department and influences regulatory expectations broadly.

HHS Strategic Plan and ONC requirements

The HHS Strategic Plan for AI establishes a framework spanning safety, equity, infrastructure, workforce development, and modernising care delivery. The Office of the National Coordinator for Health Information Technology (ONC) — administering the certified Electronic Health Record (EHR) program — has imposed AI-specific obligations on certified EHR vendors including disclosure of AI-based decision support, risk mitigation, and post-market monitoring. Certified EHR vendors must disclose and mitigate risks associated with AI-based decision support systems integrated into their products.

CMS — reimbursement and coverage

The Centers for Medicare & Medicaid Services (CMS) influence AI adoption through reimbursement decisions. CMS continues to explore reimbursement pathways for AI-augmented services through coding and coverage decisions in the 2026 Medicare Physician Fee Schedule. CMS recognises that many AI tools are iterative, continuously updated services rather than one-time products, and is developing appropriate payment models. AI tools demonstrating clinical benefit can secure reimbursement, creating economic incentive for compliant deployment.

State-level healthcare AI laws

State-level activity is where most current healthcare AI regulation lives. Four 2026 themes dominate state legislation:

Mental health chatbots. California's law effective 1 January 2026 requires chatbots to clearly identify themselves as AI and bans chatbots that lack protocols to prevent content around suicide or suicidal ideation. Ohio has introduced legislation prohibiting AI from making diagnoses or therapeutic decisions and preventing AI from determining patient mental or emotional state.

Patient disclosure and consent. California AB 489 (effective 1 January 2026) prohibits developers and deployers of AI systems from using terms, letters, phrases, or design elements that indicate or imply the AI possesses a healthcare license. California AB 3030 and SB 1120 (effective January 2025) established baseline patient disclosure requirements. Texas Responsible AI Governance Act (TRAIGA), signed June 2025 and effective 1 January 2026, includes specific disclosure requirements for licensed healthcare practitioners.

Preventing AI presenting as clinical providers. Multiple states have introduced legislation restricting AI's ability to appear as a healthcare provider, requiring clear AI disclosure in patient interactions.

Payor use of AI. States are increasingly regulating AI use by health insurance payors in determining medical necessity and prior authorization. 2026 has focused on payor use of AI to downcode claims. New York, Texas, California, and others have introduced legislation requiring physician oversight, appeal rights, and transparency in AI-driven coverage determinations.

Practical compliance — what healthcare organisations must build

An effective US healthcare AI governance programme should include:

AI inventory. Every AI system used in the organisation — clinical decision support, ambient scribing, predictive analytics, payor-facing tools, patient-facing chatbots — catalogued with purpose, vendor, regulatory classification, and risk level.

Local validation. Generic vendor validation is insufficient. Healthcare AI must be validated within the specific deployment context (patient populations, clinical workflows, operational environments) before clinical implementation. Local validation is required for ongoing safety and is non-negotiable for malpractice defence.

Business Associate Agreements with AI vendors. Standard BAAs do not address AI-specific risks. Updated BAAs should cover training data use, model updates, AI-specific incident notification, audit rights, and data return/deletion on termination.

Bias monitoring. AI must be monitored for performance across demographic groups. New HHS and FDA policies on safety and post-market surveillance address bias concerns; provider compliance requires ongoing measurement, not one-time validation.

Patient consent and disclosure. Patient disclosure of AI involvement is increasingly required. For ambient scribes specifically, explicit patient consent before recording is standard; for AI-driven clinical decisions, disclosure is best practice and required in California, Texas, and other states.

Clinical governance integration. AI governance must integrate with existing clinical governance (medical staff bylaws, peer review, quality and safety committees). The Joint Commission and Coalition for Health AI (CHAI) have developed responsible AI in healthcare frameworks; a voluntary AI certification program from these bodies is planned for 2026.

Vendor due diligence. Before procuring AI tools: verify FDA clearance/approval status for any AI making clinical decisions; review evidence of bias testing; obtain training data documentation; confirm BAA terms; review state-specific disclosure compliance.

Primary sources: FDA — AI/ML-Enabled Medical Devices · HHS AI Strategy · Manatt Health AI Policy Tracker