Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disagreement among frontier LLMs on real-world fact-checks

lenz.io

Post reply on HN
Print view View original

101–110 of 377 posts

Re: Disagreement among frontier LLMs on real-world fact-checks

#101

Post by wongarsu »

One fun example: "Ruskin Bond was born on May 19, 1934, in Kasauli, Himachal Pradesh, India". Opus and Gemini believe this to be true, GPT 5.4 believes it's false, Sonar thinks it's mostly true. Disagreement value of 3, you can't disagree more than some models thinking it's true, some thinking it's false

But my impression from 2 minutes on Wikipedia is that the most likely disagreement is on the "Himachal Pradesh, India" part. The guy was born on that date, in that town. But while the town is today in the state of Himachal Pradesh in India, that was not true in 1934. When he was born, the city was in the Punjab States Agency of the British Raj.

So was he born in Himachal Pradesh, India or not? I find both True and False equally defensible here

https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...

https://en.wikipedia.org/wiki/Ruskin_Bond

Re: Disagreement among frontier LLMs on real-world fact-checks

#102

Post by vjvjvjvjghv »

simonw wrote: post #11

Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…

"Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers."

That's exactly the stupidity of the public discourse these days. People feel compelled to take a clear position although there is much more subtlety in many issues. It's not ok to say "I don't know", "it depends" or "as far I know". And then people feel they need to defend this position no matter what new information comes up.

Re: Disagreement among frontier LLMs on real-world fact-checks

#103

Post by jawns »

"Extraterrestrial life exists somewhere in the universe."

GPT-5.4: Misleading

Opus 4.7: Misleading

Gemini 3: FALSE

Gemini 3 (Retrieval): FALSE

Sonar Pro: FALSE

It's a weird fact claim, because the ground truth is "nobody knows for sure" and that's not one of the available options.

Re: Disagreement among frontier LLMs on real-world fact-checks

#105

Post by thfuran »

feanaro wrote: post #77
simonw wrote: post #11

Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…

> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer? Am I missing something? [1]: https://en.wikipedia.org/wik…

It’s misleading because it’s false. But yes, I think false is quite plainly the better answer there.

Re: Disagreement among frontier LLMs on real-world fact-checks

#106

Post by kostaj »

ipunchghosts wrote: post #4

I think ppl only care about how Claude or codex does.

GPT-5.4 and Opus 4.7, specifically, agree between themselves on 65% of the claims - 95% CI 62–68%. I.e., in at least 35% of the claims, one of the two models is wrong under this 4-bucket rubric.

Re: Disagreement among frontier LLMs on real-world fact-checks

#108

Post by kevin_thibedeau »

falcor84 wrote: post #80
pjc50 wrote: post #39

Earlier quoted context omitted.

If you can consistently construct "true but misleading" content, you may be qualified to work at a major newspaper.

> true but misleading It seems to me that for many newspapers the bar is now significantly lower, at something like "not quite entirely untrue"

Allegedly.

Re: Disagreement among frontier LLMs on real-world fact-checks

#110

Post by hombre_fatal »

feanaro wrote: post #77
simonw wrote: post #11

Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…

> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them". The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer? Am I missing something? [1]: https://en.wikipedia.org/wik…

Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.
Post reply on HN