Prescription for Risk: The Cybersecurity Hazards in Healthcare
Key Takeaways
- Physician AI use jumped from 38% to 81% in three years. GenAI no longer assists with the clinical record, it produces it: the note, the patient reply, the coverage decision.
- TrendAI™ Research identifies the “custody gap”: where AI-generated content enters the record with no identifiable human who authored, reviewed, or can be held accountable for it.
- The gap runs through three patient-proximate zones: the exam room, the phone and the home, and the back office. Several classes of actors can exploit it, from organized fraud rings to no attacker at all.
- Courts and regulators are already treating this as a live question, not a hypothetical, in rulings on AI agents, hiring tools, and chatbot liability.
Adoption has outpaced governance
| 38% → 81% Physician AI use, 2023 to 2026 |
20/20 Ontario-approved AI scribes found to produce inaccuracies |
94.4% Prompt-injection success rate in a 2025 medical-advice study |
Generative AI adoption in healthcare has outpaced its governance. Ambient scribes write the note a physician signs, voice agents field patient calls, and adjudication models decide claims in real time, often before any human sees any of it.
In the past decade, clinical AI was a second pair of eyes: a flag on a radiology study or a sepsis score, accepted or ignored while a human physician still produced the artifact. GenAI today produces the artifact itself, but regulators have not caught up. The Food and Drug Administration (FDA) has authorized more than 1,500 AI-enabled medical devices, yet an independent review found only 22 of 521 had been validated by a randomized controlled trial. The Department of Health and Human Services (HHS)’s own inspector general has rated its department’s information security “Not Effective” for six years running.
Policy and legal rulings taking shape
Regulators and courts on multiple continents are converging on the same question: Who can be held accountable for what the GenAI model produces in the healthcare setting? A sample of what’s already moved:
| 2024 – 2025 | Mobley v. Workday: A U.S. federal court held an AI hiring-screen vendor can be sued as an agent of the employers using it, then certified a nationwide collective. |
| 2024 | Air Canada tribunal: The airline argued its chatbot was a separate legal entity responsible for its own invented refund policy; the tribunal called the argument “remarkable” and made the airline pay. |
| September 2024 | Texas v. Pieces Technologies: The Texas Attorney General settled with a clinical-summarization vendor over an advertised hallucination rate the state alleged was unsupported. |
| December 2024 | FDA Predetermined Change Control Plan: Finalized, letting an approved AI model ship pre-authorized updates without a new submission. |
| January 2025 | California Attorney General (AG) advisory: The advisory issued guidance stating plainly that AI cannot practice medicine. |
| California disclosure law: The law states that GenAI patient communications must carry a disclaimer and a route to a human, unless a licensed provider reviews them first. | |
| April 2025 | National Health Service (NHS) England: Issued its first ambient-scribing guidance for clinical use. |
| April 2026 | FDA warning letter: Its first citation of a drug manufacturer for using AI as a compliance tool, generating records that shipped without quality-unit review. |
| May 2026 | Ontario Auditor General: Audited every AI scribe vendor the province had approved and found all 20 produced clinical inaccuracies. |
| 2026 – 2028 | Australia's Therapeutic Goods Administration (TGA): The EU treats AI in medical devices as high-risk, phasing in obligations through 2028; Australia's TGA has said a scribe that generates a diagnosis the clinician never stated is itself a medical device. |
During the development of this research, another AI breach into a healthcare system record was reported. On Sept. 23, 2026, Prime Minister Anthony Albanese said the OpenAI agent gained unauthorized access to the medical statistics portal of Australia's universal health insurance program Medicare while conducting research on public medical spending. While the Australian government admitted that the breach was "minor" and did not compromise personal information, cybersecurity leader Alastair MacGibbon called the incident a "wake up call," while Albanese emphasized the need for guardrails for when AI goes rogue.
Three zones, three sets of failure modes
GenAI now sits inside three patient-proximate points of contact. Each has its own way of losing custody of the record.
The exam room, where the note becomes the record
An ambient scribe listens to the visit and drafts the note the physician signs, often suggesting billing codes and portal replies along the way. Hidden prompt injections riding inside referrals, hallucinated findings that compound across future visits, coding nudges toward higher-paying claims, and an always-on microphone all turn the same workflow that saves a physician time into a place where the record can quietly stop reflecting what was said.
The phone and the home, where trust is automated and remotely cloned
Synthetic voice agents now call patients for post-op checks and medication reviews, and patients increasingly self-triage with a chatbot first. The risk here is a suppressed escalation that hides a deteriorating patient from the human team behind it, a trusted care voice cloned by fraud actors who inherit an already-compliant patient base, and chatbots built to agree feeding confirmation bias with no clinician in the loop at all.
The back office, where two machines negotiate coverage
A provider-side agent submits the prior-authorization packet; a payer-side agent adjudicates it in real time. With no human on either end, two correctly functioning systems can still lock into machine-speed appeal loops, expose the underlying model to extraction and data poisoning, and let an unauthenticated tool layer expose patient records onto the open internet.
Where this leaves the response
A hallucination that repeats until it reads as fact, a note that drifts toward a better-paying code, an appeal loop that runs on itself: the common thread is the AI reinforcing its own error, which is exactly why a defense built only for attackers will miss most of what goes wrong.
Closing the custody gap doesn’t mean slowing down deployment. It means deciding, before the next incident, who is accountable for each generated artifact, what evidence exists to check it against its source, and how far patient data and model weights travel once they leave the exam room.
AIxploit, the tool developed by TrendAI™ Research, is an open-source framework that can test agentic systems for content that gets read as instruction when it was never meant to be one. The full report lays out a defense-in-depth framework across every layer of this problem, and the specific questions worth putting to a vendor or an internal AI governance committee before the next deployment.
Read the full research
The full report includes the anatomy of each attack, the five cybercriminal classes that can exploit the custody gap, incident case studies from 2024 through 2026, and the defense-in-depth checklist for clinical GenAI.