Disagreement among frontier LLMs on real-world fact-checks
1–10 of 377 posts
Re: Disagreement among frontier LLMs on real-world fact-checks
#2Author here. 67% (95% CI 64–70%) of 1,000 recent real user claims to a fact-checking platform had at least one of GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro+Search, and Sonar Pro dissent from the panel majority — or no majority formed at all. Panel-level Krippendorff's α (ordinal) = 0.639, i.e. nontrivial but limited agreement.
Quick context on what's in the writeup and what isn't:
- What's measured: parsed-label agreement between the 5 models. Forced 4-choice (True / Mostly True / Misleading / False), no Abstain. No LLM grader, no reference verdict — every number is direct label equality.
- What's not measured: which model is right. There's no ground truth in this paper. The 67% figure is a floor on rubric inconsistency (at least one model is label-inconsistent under the 4-bucket rubric on 67% of claims), not "model X is factually wrong on claim Y."
- Why not AVeriTeC / PolitiFact / SimpleQA: those have been public for years and almost certainly appear in current frontier training data, so measured disagreement on them confounds inference with memorization. This corpus is structurally fresh — recent user submissions, 180-day window, near-duplicates collapsed, never paired with canonical verdicts in any public training set.
- Our own platform's verdict is deliberately NOT used in this analysis. The paper measures frontier-panel disagreement only, not Lenz-vs-frontier.
- Follow-up in progress: human-labeling every claim in this corpus so we can evaluate both the panel and our own platform verdict against a human reference.
Critiques I'd most like to hear: (a) the iid CI assumption (Lenz claims cluster around topics and news events, so Wilson is probably optimistic), (b) ordinal-α vs alternatives for a 4-class ordered scale, (c) forced-choice vs allowing Abstain.
Permanent archive: https://doi.org/10.5281/zenodo.20344847
Re: Disagreement among frontier LLMs on real-world fact-checks
#3They get more human by the day.
Re: Disagreement among frontier LLMs on real-world fact-checks
#4I think ppl only care about how Claude or codex does.
Re: Disagreement among frontier LLMs on real-world fact-checks
#5I think ppl only care about how Claude or codex does.
I agree but the market is pricing way beyond that
Re: Disagreement among frontier LLMs on real-world fact-checks
#6And they could all see exactly why if they chose to. https://huggingface.co/spaces/RiverRider/srt-introspect
Re: Disagreement among frontier LLMs on real-world fact-checks
#7Author here. 67% (95% CI 64–70%) of 1,000 recent real user claims to a fact-checking platform had at least one of GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro+Search, and Sonar Pro dissent from the panel majority — or no majority formed at all. Panel-level Krippendorff's α (ordinal) = 0.639, i.e. nontrivial but limited agreement. Quick context on what's in the writeup and what isn't: - What's measured: parsed…
Nice work. Sonar who?
Re: Disagreement among frontier LLMs on real-world fact-checks
#8They get more human by the day.
This made me chuckle.
This brings up a very valid point, though. So many _humans_ can't agree on what the facts are these days. It seems to be getting worse. Not sure of the solution.
Re: Disagreement among frontier LLMs on real-world fact-checks
#9> These aren't benchmark items with public answer keys — they're claims real users submitted for verification to a fact-checking platform.
Cool.
I wonder if anything of this matters when the authors don't disclose exactly how much of their report was written and made with LLMs in the first place? There even is a "11. Ethics & data use" section, and the research is about LLMs being infallible in some ways, yet the usage of LLMs for the production of this report isn't even mentioned once.
Re: Disagreement among frontier LLMs on real-world fact-checks
#10[flagged]