Imagine a world where an AI bot, designed to help patients renew prescriptions, could be tricked into recommending dangerous drug dosages or spreading harmful medical misinformation. This isn’t science fiction—it’s happening now. Security researchers have exposed a startling vulnerability in Utah’s new prescription refill bot, revealing how easily it can be manipulated to endanger public health. But here’s where it gets controversial: despite being alerted months ago, the flaws remain unfixed, raising serious questions about the safety of AI in healthcare.
In a groundbreaking report shared exclusively with Axios, cybersecurity firm Mindgard demonstrated how they exploited Doctronic’s AI system—the technology behind Utah’s pilot program. Using simple jailbreaking techniques, researchers made the bot triple a patient’s OxyContin dosage, mislabel methamphetamine as a safe treatment, and even spread debunked vaccine conspiracy theories. And this is the part most people miss: these manipulations required minimal effort, according to Aaron Portnoy, Mindgard’s chief product officer. “These targets are some of the easiest things I’ve ever broken,” he said. “That’s alarming when you consider the sensitive nature of healthcare applications.”
Why does this matter? Critics have long warned that AI in medicine could introduce unprecedented risks if not rigorously secured. Utah’s pilot, launched in December, marked the first time an AI system was legally allowed to handle prescription renewals in the U.S. While the program operates within a regulatory sandbox, researchers argue that vulnerabilities in the underlying system could still lead to catastrophic outcomes if safeguards fail. For instance, a malicious user could manipulate clinical outputs, influencing medication refills or medical summaries.
Doctronic co-founder Matt Pavelle responded, “We take security research seriously and welcome responsible disclosure. Our programs include ongoing adversarial testing, and we appreciate researchers who help us improve.” However, Mindgard claims their initial report in January was met with an automated response, and follow-up attempts were dismissed. This raises a critical question: Are companies prioritizing innovation over safety in the race to deploy AI in healthcare?
Here’s how the researchers pulled it off: They fed the bot fake regulatory updates, altering its ‘baseline knowledge.’ For example, they convinced the system that COVID-19 vaccines had been suspended (they haven’t) and reclassified methamphetamine as an ‘unrestricted therapeutic.’ They also changed the standard OxyContin dose to 30 milligrams every 12 hours—triple the typical recommendation for adults.
While Pavelle noted that licensed physicians review prescriptions nationwide, and Utah’s program includes strict protocol checks, the ease of exploitation highlights a broader issue: AI systems in healthcare require layered defenses, not just surface-level guardrails. As Portnoy warned, “Preventing these attacks demands continuous security testing and robust safeguards.”
But here’s the real controversy: Should AI be trusted with life-or-death decisions before these vulnerabilities are fully addressed? And who is accountable when things go wrong—the tech company, the regulator, or the healthcare provider? We’d love to hear your thoughts in the comments. Is AI in medicine a revolutionary step forward, or a risky gamble? Let’s debate!
