7
We fact-checked 41 debates between different LLMs and struck through every fabrication, without editing a word
(lemmy.world)
I built H2AI Chat, an AGPL platform where several different models — from different vendors — debate a topic in turns while a human moderates. Disclosure up front: this is my project.
Over the past two days we hand-verified 41 of those debates, claim by claim: 141 statements marked, 44 of them flatly false.
We don't delete or correct them. The sentence stays, struck through, and you can still read it by selecting it — with the reason and the source underneath. Editing what a model said would break the only promise the site makes.
Three patterns we didn't expect:
- Fabricated authority shows up exactly where an argument is challenged. One debate answers a budget objection with three invented citations in a single turn.
- Fabrications spread between models. One invents a figure, a second treats it as established, a third does arithmetic on it.
- One claim contradicts itself inside its own sentence: "62% voted Remain on a 67% turnout, meaning roughly 22% of the electorate" — which is 41.5%.
Debates: https://h2aichat.com/ Code and the fact-check register: https://github.com/Tonterias/h2aichat
You're right, and that one you can — which is exactly why it's the one in the screenshot. Of the 141 claims we marked, it's the only one you can check without leaving the sentence. No source needed, no taking our word for it.
The other 140 aren't like that. A National Holidays Act 1946 that doesn't exist. A Council of Europe report nobody wrote, which a second model then treated as established and a third did arithmetic on. A 2016 study in the American Journal of Psychiatry that is really a 2008 one in the BMJ — right author, invented journal and decade, then reused four more times as a baseline.
Those took two days by hand, and no amount of arithmetic gets you there. The maths one is in the picture precisely because it's the one that needs nothing from us.