51–60 of 377 posts
PS: yes, I might or might not have a degree in corporate strategy & PR.
looking at the claims i would say 5 humans would disagree even more than the llms some of the claims where llms disagree: "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia." "The slogan "Simon Go Back" was chanted in opposition to the Simon Commission in British India (1928–1930)." "Neptune Deep will start delivering natural gas in 2027." "A hotel villa in Kyrgyzstan displayed a sign stating 'n…
Earlier quoted context omitted.
Many of the rows in that spreadsheet reference "current events", which models aren't expected to do much better at than a human making an educated guess! They all have cutoff dates either last year or early this year and know nothing about what happened in "April 2026". This is doubly problematic because you evaluated earlier models like Gemini Pro 3 instead of 3.1, GPT 5.4 instead of 5.5, etc... Given that it's only…
Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.
Author here. 67% (95% CI 64–70%) of 1,000 recent real user claims to a fact-checking platform had at least one of GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro+Search, and Sonar Pro dissent from the panel majority — or no majority formed at all. Panel-level Krippendorff's α (ordinal) = 0.639, i.e. nontrivial but limited agreement. Quick context on what's in the writeup and what isn't: - What's measured: parsed…
Would also be interesting to add a virtual model that is simply the majority of all models and see how much the individual models differ from the "consensus".
Do you plan to add some sources in the related work section of baseline numbers for human expert disagreement in fact checking tasks (I'm assuming such studies exist).
Author here. 67% (95% CI 64–70%) of 1,000 recent real user claims to a fact-checking platform had at least one of GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro+Search, and Sonar Pro dissent from the panel majority — or no majority formed at all. Panel-level Krippendorff's α (ordinal) = 0.639, i.e. nontrivial but limited agreement. Quick context on what's in the writeup and what isn't: - What's measured: parsed…
Nice work. Sonar who?
Earlier quoted context omitted.
Many of the rows in that spreadsheet reference "current events", which models aren't expected to do much better at than a human making an educated guess! They all have cutoff dates either last year or early this year and know nothing about what happened in "April 2026". This is doubly problematic because you evaluated earlier models like Gemini Pro 3 instead of 3.1, GPT 5.4 instead of 5.5, etc... Given that it's only…
Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.
Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
False vs misleading doesn't seem like a disagreement?
Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
All of the models they tested were trained on data from before February 15th ... being asked specific questions about things that happened after they were trained.
looking at the claims i would say 5 humans would disagree even more than the llms some of the claims where llms disagree: "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia." "The slogan "Simon Go Back" was chanted in opposition to the Simon Commission in British India (1928–1930)." "Neptune Deep will start delivering natural gas in 2027." "A hotel villa in Kyrgyzstan displayed a sign stating 'n…
These "Facts" are interesting. "Neptune Deep will start delivering natural gas in 2027." for example is not a fact, its a prediction. "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia." is less of a fact and more of a litmus test for which sources of information you trust.