FDA Chief Calls for 'Shared Accountability' in Healthcare AI Regulation

FDA Chief Calls for 'Shared Accountability' in Healthcare AI Regulation

Warns of AI Hallucination Risks

In a wide-ranging interview with the Journal of the American Medical Association (JAMA), Food and Drug Administration (FDA) Commissioner Dr. Robert Califf outlined his vision for regulating artificial intelligence (AI) in healthcare, emphasizing that oversight cannot rest solely with the FDA. Despite having approved nearly 1,000 AI-enabled medical devices, Califf stressed that the rapidly evolving nature of AI technology requires a new regulatory approach involving healthcare systems, technology companies, and medical journals working in concert.

Unlike traditional medical devices or drugs that remain static after approval, Califf compared AI algorithms to intensive care unit (ICU) patients requiring constant monitoring. "Think of an AI algorithm like an ICU patient being monitored as opposed to drugs and devices in the old-fashioned way," he said, noting that AI systems change based on their inputs and societal factors. This dynamic nature presents unique regulatory challenges that the FDA's traditional approval process wasn't designed to address.

The commissioner addressed several specific AI technologies, from sepsis prediction models to artificial intelligence embedded in consumer devices like the Apple Watch. Of particular concern are large language models (LLMs), which Califf noted are especially problematic due to their lack of transparency and potential for "hallucinations" - generating false or fabricated information. While he suggested using multiple AI models to cross-check each other's outputs as one potential safeguard, notably absent from the discussion were specific requirements for human oversight or detailed supervision protocols for these systems in clinical settings.

 
A particular concern Califf highlighted is the current use of AI in healthcare systems primarily to optimize finances rather than health outcomes. He warned that without proper oversight, AI systems could exacerbate healthcare disparities by catering to patients with better insurance or financial resources. While the FDA focuses on safety and effectiveness, Califf acknowledged the agency lacks authority to consider economic factors or mandate comparative effectiveness studies.

Looking to the future, Califf indicated that the FDA is seeking additional Congressional authority to set standards for post-market monitoring of AI systems. However, he emphasized that even with expanded authority, the FDA cannot monitor every AI implementation in clinical practice. Instead, he envisions a system of mutual accountability where healthcare providers, professional societies, and other stakeholders play active roles in ensuring AI systems perform as intended.

The commissioner's comments come at a crucial time as healthcare AI adoption accelerates. With the technology becoming increasingly embedded in clinical practice, from drug development to clinical decision support algorithms, Califf's call for shared accountability suggests a significant shift in how medical AI might be regulated, moving away from the traditional model of singular FDA oversight toward a more collaborative, ecosystem-based approach to ensuring safety and effectiveness.

Gaps in the Interview - Data Privacy Questions Loom Large in FDA's AI Healthcare Push

Following Food and Drug Administration (FDA) Commissioner Dr. Robert Califf's recent discussion of artificial intelligence (AI) regulation in healthcare, notable gaps remain regarding the use of electronic medical records (EMRs) in AI development and associated privacy concerns. While Califf outlined broad regulatory frameworks in his Journal of the American Medical Association (JAMA) interview, critical questions about patient data protection remain unaddressed.

Key unaddressed issues include how the FDA will approach the use of patient records in training large language models (LLMs), compliance with the Health Insurance Portability and Accountability Act (HIPAA), and requirements for patient consent. As healthcare systems increasingly adopt AI tools trained on medical records, the absence of clear guidance on these matters becomes more pressing.

The FDA's focus on post-market monitoring and preventing AI bias is important, but we need equal attention on protecting patient privacy during the development phase. Healthcare systems are sitting on vast troves of sensitive patient data that AI companies are eager to access.

Several critical questions remain for the FDA to address:

• - What standards will govern the use of EMRs in training healthcare AI systems?
• - How will patient consent be handled for the use of medical records in AI training?
• - What safeguards will be required to prevent the extraction of personal health information from AI models?
• - How will HIPAA compliance be assured when using AI systems trained on patient records?
• - What role will the FDA play in monitoring data privacy alongside its focus on AI safety and effectiveness?

The privacy concerns become particularly relevant given Califf's comments about AI systems constantly changing based on new inputs. This dynamic nature raises questions about how patient data privacy will be maintained as systems evolve and learn from new medical records.

These privacy and data security considerations will likely need to be addressed as part of the "shared accountability" framework Califf described, requiring collaboration between the FDA, healthcare providers, technology companies, and privacy experts to establish appropriate guidelines and safeguards.

While the potential applications of LLM such as ChatGPT are undeniably exciting, it's vital to discuss the topic of HIPAA compliance and how it protects patient health information (PHI). Given the sensitive nature of health data, any tool used within the healthcare system must ensure the secure handling of PHI.

Summary of Interview

Here's a summary of the key points from the JAMA interview with FDA Commissioner Dr. Robert Califf about AI regulation in healthcare:

1. Current State and Context:

• - The FDA has already approved nearly 1,000 AI-enabled devices
• - AI is becoming deeply integrated into healthcare, from devices to drug development and supply chains
• - The FDA is taking a proactive approach to regulation while trying to balance innovation

2. Regulatory Approach:

• - FDA can't monitor every AI implementation directly, similar to how they don't inspect every farm
• - Focus is on creating "guardrails" and safety mechanisms to guide industry
• - Special emphasis on continuous monitoring of AI algorithms after deployment, comparing them to "ICU patients" that need ongoing monitoring
• - Unlike traditional drugs/devices, AI systems change based on inputs and societal factors

3. Key Challenges:

• - Many health systems are currently using AI primarily to optimize finances rather than health outcomes
• - There's limited authority for FDA to regulate post-market performance
• - Language models present particular challenges due to lack of transparency and potential for "hallucinations"
• - Need to balance the interests of both large companies and small startups

4. Shared Responsibility:

• - Califf emphasizes that regulation requires collaboration between FDA, healthcare systems, professional societies, and medical journals
• - Health systems and clinicians need to demand transparency about AI performance metrics
• - Need for continuous monitoring of AI algorithms' performance in real-world settings

5. Future Needs:

• - FDA would benefit from additional Congressional authority to set standards for post-market monitoring
• - Need for better systems to monitor AI performance after deployment
• - Importance of ensuring AI doesn't exacerbate healthcare disparities
• - Need for new approaches to clinical trials and evaluation methods for AI systems

6. Current Limitations:

• - FDA's regulatory authority is primarily focused on safety and effectiveness, not economics
• - Cannot require comparative effectiveness data
• - Limited ability to monitor algorithms being used by health systems
• - Cannot directly regulate every AI implementation in clinical practice

The interview emphasizes the need for a collaborative ecosystem approach to AI regulation in healthcare, rather than relying solely on FDA oversight. 

AI Technologies

Looking at the interview content specifically regarding AI technologies and hallucinations:

AI Technologies Discussed:

1. Sepsis prediction models - Used as a specific example of AI requiring monitoring
2. Language models/Large Language Models (LLMs) - Discussed as particularly challenging for regulation
3. Decision support algorithms - Mentioned in context of clinical implementation
4. AI in drug development and discovery - Referenced but noted as primarily industry's domain
5. AI embedded in consumer devices (e.g., Apple Watch) was mentioned

Regarding LLM Hallucinations and Human Supervision:

• - The topic of hallucinations was briefly mentioned but not extensively discussed
• - Califf suggested using one large language model to check another as a potential safeguard against hallucinations
• - He specifically mentioned hallucinations in the context of clinical notes, noting it "Could be a big problem"
• - However, the interview didn't deeply explore the need for human supervision or specific oversight requirements for LLMs

A notable gap in the interview was that while hallucinations were acknowledged as a concern, there wasn't detailed discussion about:

• - Specific requirements for human oversight
• - How to implement supervision in clinical settings
• - What role clinicians should play in monitoring AI outputs
• - Specific safeguards against LLM hallucinations beyond the suggestion of using multiple models

The interview focused more on broad regulatory frameworks and ecosystem-wide accountability rather than detailed discussions of specific AI technologies or supervision requirements. 

FDA Commissioner Robert Califf on Setting Guardrails for AI in Health Care | Digital Health | JAMA | JAMA Network

Roy Perlis, MD, MSc^1,2,3,4; Jennifer Abbasi⁵

Author Affiliations Article Information

JAMA. Published online November 22, 2024. doi:10.1001/jama.2024.24760