FDA rushed out agency-wide AI tool—it’s not going well

The Food and Drug Administration (FDA) has launched an agency-wide artificial intelligence tool named Elsa, aiming to enhance performance and optimize processes from scientific reviews to inspections. Officials claim Elsa will accelerate clinical protocol reviews, shorten scientific evaluations, and identify high-priority inspection targets. FDA Chief AI Officer Jeremy Walsh heralded the launch as the \"dawn of the AI era at the FDA,\" stating it's already enhancing every employee's potential.

Notably, FDA Commissioner Marty Makary emphasized the tool's rapid deployment, proudly announcing the rollout was ahead of schedule and under budget, thanks to internal collaboration. Elsa is based on Anthropic's Claude LLM and was developed by Deloitte within a secure GovCloud environment, designed to keep all information strictly within the agency.

However, despite the official enthusiasm, the rollout is reportedly facing significant issues. According to NBC News, FDA staff testing Elsa found it provided inaccurate summaries of public information. Furthermore, FDA staffers speaking with Stat news described the tool as \"rushed,\" \"buggy,\" and \"overhyped.\" They cautioned that in its current form, Elsa should be limited to administrative tasks and not used for critical scientific work.

Some employees feel the FDA leadership and external partners have an inflated view of AI's capabilities, believing it can replace staff and cut review times, which staff believe it cannot. There are also concerns about the lack of established guardrails and policies for the tool's use amidst the hurried deployment.

This situation highlights the potential pitfalls of rapidly deploying complex AI systems without sufficient testing, staff training, and careful consideration of policy and accuracy, particularly in critical public health agencies like the FDA.