A preprint posted to medRxiv on 28 October had 15 large language models review the same three clinical trial papers that 607 human reviewers assessed in 2008 — papers deliberately altered to contain methodological errors. The result is uncomfortably clean: the best model, Anthropic's Claude Opus 5.5, found 96% of the planted errors. The worst found 53%. The humans found 27%.

The caveats are part of the story. The 2008 reviewers were not asked to list every flaw; they were judging whether a manuscript should be published, and some stopped once they found an error serious enough to reject it. A model never stops. The standards for clinical trials now in use were also not yet widely applied in 2008, and the doctored versions have been public since then.

So Ruslan Rust's team at the University of Southern California ran a second test: 58 more doctored clinical studies from this year, most published after the models' training, with no internet access and no list of the inserted errors. The models did well there too. On the undoctored versions of the same manuscripts they flagged fewer errors — but still some, including in papers published in JAMA and the New England Journal of Medicine.

The most frequent complaint was conclusions not supported by the results. That is not always a factual error, but it is a problem.

The practical obstacle is political rather than technical: many medical journals bar their human reviewers from using AI. Anyone who wants to use these results has to change those rules — and decide which parts of review can be automated and which cannot.