[flagged]
Disagreement among frontier LLMs on real-world fact-checks
151–160 of 377 posts
Re: Disagreement among frontier LLMs on real-world fact-checks
#152Earlier quoted context omitted.
Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.
The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) and it does indeed lack the qualitative aspect.
Sure, but people are drawing conclusions beyond "LLMs said different words" and trying to use it to analyze whether LLMs were wrong about the underlying facts, but that information isn't available to us.
Re: Disagreement among frontier LLMs on real-world fact-checks
#153Earlier quoted context omitted.
But the models are more intelligent than humans already and sentient beings, right? So they shall know the meanings innately. So, you don’t need to explain them what they mean. You may give them better instructions, but they should already have the intellect to understand the assignment. Right, right?
> But the models are more intelligent than humans already and sentient beings, right? Only if you listen to charlatans.
True. If you didn't know my stance on AI already, here's a primer :) [0].
IOW, that comment was a sarcastic poke from someone who already supports AI workloads at work and have some knowledge about how all this works. ;)
[0]: https://notes.bayindirh.io/notes/Lists/Discussions+about+Art...
Re: Disagreement among frontier LLMs on real-world fact-checks
#154Earlier quoted context omitted.
If you can consistently construct "true but misleading" content, you may be qualified to work at a major newspaper.
> true but misleading It seems to me that for many newspapers the bar is now significantly lower, at something like "not quite entirely untrue"
Almost, but not entirely, quite unlike the truth.
Re: Disagreement among frontier LLMs on real-world fact-checks
#155More interesting part probably worth highlighting: The SAME model won't always return the same output when prompted with the same fact check.
You ask a human 1000 times a fact check question, they say the same answer 1000 times. You ask an LLM the same question a 1000 times, your results could vary significantly.
Humans work based on the Metamemory (knowing what they know), while LLMs are picking from statistical probability.
Re: Disagreement among frontier LLMs on real-world fact-checks
#156Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
This is a great example of why prompt engineering is still relevant. Without providing definitions and examples and a well defined rubric, you’re going to see different models disagree by a level in either direction. When you get more prescriptive the models tend to agree better.
I’ve experimented with AI grading for undergraduate math courses, and see basically the same thing. If you just tell the AI “grade this problem and assign a letter grade” then I’ve only seen about 30% agreement between a human assigned grade and the AI assigned grade. But over 75% agreement if you say a “match” is within one letter grade. And to get better agreement you have to spend a lot more time on the rubric- what kinds of mistakes are a big deal, what kinds of mistakes are not a big deal, how much work is required to be shown to get credit, a couple examples of each letter grade. Once you have done that, the AI gets a lot better agreement with human graders, but it is hard to know when you’ve given enough guidance for a problem.
Re: Disagreement among frontier LLMs on real-world fact-checks
#157Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
The examples seem intentionally diverse, but I haven't seen one that I would be surprised for someone to post about in the format of "ChatGPT/Gemini/Claude/Qwen/... says:"
So the examples are good, I think. The rest is philosophy.
The links you posted only show a frozen loading spinner for me (iOS Safari).
(I looked at the csv in Numbers instead)
Re: Disagreement among frontier LLMs on real-world fact-checks
#158"Extraterrestrial life exists somewhere in the universe." GPT-5.4: Misleading Opus 4.7: Misleading Gemini 3: FALSE Gemini 3 (Retrieval): FALSE Sonar Pro: FALSE It's a weird fact claim, because the ground truth is "nobody knows for sure" and that's not one of the available options.
Isn't misleading the correct option here then?
False makes sense if you are interpreting it strictly as "has this been proven?"
Re: Disagreement among frontier LLMs on real-world fact-checks
#159That's better than all agreeing on the wrong answer, however.
I've had multiple models give the same wrong answer or even fabricate the same nonexistent reference based on a similar prompt.
My most common chatbot prompt is "X that you mentioned above doesn't seem to actually exist."
Re: Disagreement among frontier LLMs on real-world fact-checks
#160Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
>Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected."
The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing.
So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed "among the most common reasons" or not.
That "true" and "mostly true" means effectively the same thing is irrelevant. It could just as well trip me up, and I'm a human. If somebody told me either answer, I'd still consider them right if the basic fact was right.