What's really weird to me is that "I don't know" is not a valid answer in this experiment while we can all agree that's the main issue with LLM right now is that they will happily "roleplay" an answer when they have nothing in their dataset corresponding to your query.
Disagreement among frontier LLMs on real-world fact-checks
161–170 of 377 posts
Re: Disagreement among frontier LLMs on real-world fact-checks
#162I’m no expert but if LLMs are token prediction machines, and you tell it to not build an explanation before the answer, isn’t it less likely that the token prediction for the final answer will have less raw material before it to build a grounded response?
In other words: no explanation > no foundation for prediction of the answer tokens?
Re: Disagreement among frontier LLMs on real-world fact-checks
#163Earlier quoted context omitted.
Without providing definitions of "True / Mostly True / Misleading / False" to each rater, I rate the article's claim that "Only one verdict bucket can be correct per claim" as false . Something can be simultaneously "misleading" and either true or false. Which category should something go in if it's "mostly false"? How much can something be wrong before it goes from "mostly true" to "false" (objectively, both have so…
>Something can be simultaneously "misleading" and either true or false. Which category should something go in if it's "mostly false"? Disagree. The definition of misleading is a true fact that is presented in a way to lead you to a false conclusion. Example: "Most good engineers are male". It is true as a consequence of most engineers being male in general, but it leads the reader to a potential false implication tha…
> but it leads the reader to a potential false implication that an average man is better than an average woman.
I think that's _you_ turning the statement into something much broader than intended. The claim is about engineers and you're jumping from "men are better than women in engineering" to "men are better overall."
To give a related example, "Most good NBA players are black." I don't think anyone would bother trying to couch this in a bunch of "well, for all we know that's just a function of more NBA players being black than white" arguments, nor would anyone be lead to think "the average black man is better than the average white man" as a result of that statement. I _do_ agree however that there are some people who see rather narrowly-defined statements and turn them into something they're not...
Re: Disagreement among frontier LLMs on real-world fact-checks
#164Dissent and consensus among frontier models is a good thing. Just like on a team of high performers, there are a million ways to skin a grape. In my research, I've found that models perform better when they operate as a collective system with reputation, incentives, and accountability instead of isolated oracles answering alone. Agreement, dissent, and correctness should all carry rewards and consequences. Just like…
Not on objective truth though. That's how you get misinformation.
Re: Disagreement among frontier LLMs on real-world fact-checks
#165Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
> Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected." The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing. So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed…
This study treats models disagreeing - returning both true and mostly true - as a failure.
Re: Disagreement among frontier LLMs on real-world fact-checks
#166I think ppl only care about how Claude or codex does.
GPT-5.4 and Opus 4.7, specifically, agree between themselves on 65% of the claims - 95% CI 62–68%. I.e., in at least 35% of the claims, one of the two models is wrong under this 4-bucket rubric.
but that's without internet search - everyone I know uses the models that search when they need to, and I'm sure GPT and Opus would agree on almost everything if 1) they searched when necessary, and 2) they were allowed to give context to their answers instead of being hamstrung to get specious "research" results.
Re: Disagreement among frontier LLMs on real-world fact-checks
#167Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
The examples seem intentionally diverse, but I haven't seen one that I would be surprised for someone to post about in the format of "ChatGPT/Gemini/Claude/Qwen/... says:" So the examples are good, I think. The rest is philosophy. The links you posted only show a frozen loading spinner for me (iOS Safari). (I looked at the csv in Numbers instead)
Weird, I'm loading them in Mobile Safari myself.
Re: Disagreement among frontier LLMs on real-world fact-checks
#168Honey does not spoil over time under normal storage conditions.,2026-02-17T04:11:51.495452+00:00,Science,True,True,True,True,Mostly True,1
If outcomes like these are collapsed on True-side then the disagreement will reduce from the headline number.
Re: Disagreement among frontier LLMs on real-world fact-checks
#169Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
> "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia" I actually don't know which way you came down on that one? I think strictly it's false but "mostly true" would be justifiable? (as in, to say it's false would be misleading if it lead the reader to assume there was no attack around that time). https://www.washingtonpost.com/world/2026/05/17/ukrainian-dr... It seems it happened Saturday 16th ove…
It's impossible to answer if you don't have a search tool, and three out of the five tested models didn't have a search tool.
Re: Disagreement among frontier LLMs on real-world fact-checks
#170Earlier quoted context omitted.
Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.
Comparing models with search tools to models without - when there's no option for "I am unable to answer this question without access to search" - doesn't make sense to me.
Agree about comparing models with and without search capabilities. Even the two models with search capabilities (Sonar Pro and Gemini) agree only on 58% of the claims.