The pitch for artificial intelligence in medicine is everywhere: models that pass licensing exams, read scans, draft a doctor's notes. A study in Nature Medicine, published June 26, ran the harder test. It put a large language model into real clinics with real patients and measured whether anyone actually got better care. The answer came back no, at least by the measure that counts most.
A trial, not a demo
Benchmark scores are cheap. This was a pragmatic, cluster-randomized trial across 16 primary care clinics in Kenya run by Penda Health. The clinical officers who see patients there were split into two groups. One used the normal electronic record. The other had an LLM assistant layered on top of it. Over about three months in 2025, the trial covered 9,691 patients and 103 clinicians.
The number that matters
The researchers tracked something patients feel, not something a model is good at: treatment failure within two weeks of a visit, judged by independent experts. It happened to 2.2 percent of patients in the AI group and 2.0 percent in the control group. Run the statistics and the two are a wash, the confidence range sitting squarely on top of “no difference.” Reviewers found no sign the AI made care less safe.
Why a flat result is worth your attention
“No effect” reads like failure, but the rigor is the point. Failure was already rare here, near 2 percent, so there was not much room to push it lower. The clinicians were trained and competent, not the novices an assistant might rescue. And a two-week failure count misses plenty an LLM could still touch, from paperwork to how a tricky case gets triaged. None of that rescues the headline. Handing a clinician a chatbot did not, by itself, improve outcomes. In a field running on confident forecasts, a clean null result is its own kind of finding.
Comments