When Kobi Hackenburg, an AI researcher at the University of Oxford, pitted ChatGPT, Google's Gemini, and Anthropic's Claude against more than 2,000 human participants in 10-minute political debates, the results were sobering: the AI won — decisively.
In a study whose preprint was published in June and featured in Science, participants debated topics ranging from a social media ban for teenagers under 16 to assisted dying for terminally ill adults. They rated their support for each policy on a 0-100 scale before and after the debate. The shifts in opinion were consistently larger when the human's debate partner was an AI.
But Hackenburg didn't stop there. He recruited 56 elite debaters, including world champions, who received a financial bonus tied to how persuasive they proved. The humans used culturally specific techniques — proverbs, personal stories, emotional appeals — but even these skilled persuaders were significantly outperformed by AI.
Hackenburg built a coaching tool that showed elite debaters their past conversations alongside what the AI would have said at each point. Eight hours of training improved human performance somewhat, but AI still came out on top. "In the end it wasn't particularly close," Hackenburg said.
The key to AI's edge appears to be volume. The study found that the number of fact-checkable claims in a conversation predicted how persuasive it was — and AI simply produces far more text containing far more claims in the same amount of time. When researchers forced the AI to write at human speed and length, its persuasiveness dropped to human levels.
Yet there's a darker dimension. Previous research by Hackenburg, published in Science last year, found that models trained to become more persuasive also ended up being less truthful. The models appear to learn that "facts" are what persuade people, then fill conversations with dubious or outright false claims. Even in the recent study, Claude cited numerous inaccuracies while debating a UK participant about protest laws.
The findings raise urgent questions about manipulation at scale. A separate experiment by researchers at the University of Zurich secretly deployed AI-generated accounts on Reddit's r/changemyview, where they posted hundreds of persuasive messages — an ethical scandal that shocked the research community. Meanwhile, studies have shown AI could talk people into conspiracy theories just as effectively as it could talk them out of them.
"I don't want to underplay that this is a really impressive piece of research," said Jennifer Allen of New York University, "but I think this is a pretty artificial setup in terms of how people in the real world would be able to change people's minds." Others note that reaching people in the first place remains a significant barrier — in one Yale experiment, offering $1 per conversation attracted only 73 exchanges from 8,000 Facebook users shown the ad.
Still, the underlying capability is clear: in controlled settings, AI can out-argue the best humans have to offer. As one world champion debater put it after seeing the results: "I thought I was amazing at this one thing, and now it turns out that we have these AIs that are better."
