181–190 of 377 posts
Earlier quoted context omitted.
Isn't misleading the correct option here then?
False makes sense if you are interpreting it strictly as "has this been proven?"
My implicit assumption is that if you fact-check the fact-check, any label other than "true" means the original fact-check is unacceptable
Earlier quoted context omitted.
Weird, I'm loading them in Mobile Safari myself.
Sorry, I didn't wait quite long enough after the last output line appeared. After a couple of seconds, the result does appear. Happened to be just within my threshold for considering it broken, because the URL bar was "finished", and the spinner doesn't spin, but the last point is probably caused by my a11y settings (prefer no animations and no autoplay).
Earlier quoted context omitted.
Better options would have been "True", "False", "Unknown" (which opinions would fall under too). That also includes an interesting assessment of how well LLMs can identify missing information. My guess is they would be a very low number of "unknown" and a much higher level of agreement (assuming equal representation). Unless the RLHF techniques have gotten better at getting an LLM to say "I don't know", which I doubt…
Tried initially with a fifth bucket, Abstain. It was actually heavily used by some of the models. But it felt as if they are using this to "avoid" some of the hard questions, and we dropped this bucket to force them to provide a verdict.
Here's the prompt they used: Classify this claim as of : " " Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers. The claims look like this: https://lenz.io/research/llm-disagreement/data.csv I put that in Datasette Lite to make it easier to explore. Here's an example of a disagreement: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... The claim was "All alm…
Earlier quoted context omitted.
It's all fairly lazy to a degree that is mildly confusing. I also feel this among other issues would have become obvious if they had bothered to include a human fact checker baseline (i.e. asked multiple human fact checkers the same questions).
I do not think it is "lazy". Those labels are ones that human fact-checkers have been using for a decade or more. I think those human fact-checkers use those terms knowing full well that there is overlap and ambiguity between them. So I think this study ends up mixing three effects: how LLMs interpret the claims as statements about the world, how LLMs reduce that to a four-category judgment, and the inherent ambiguit…
Earlier quoted context omitted.
> "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia" I actually don't know which way you came down on that one? I think strictly it's false but "mostly true" would be justifiable? (as in, to say it's false would be misleading if it lead the reader to assume there was no attack around that time). https://www.washingtonpost.com/world/2026/05/17/ukrainian-dr... It seems it happened Saturday 16th ove…
It's impossible to answer if you don't have a search tool, and three out of the five tested models didn't have a search tool.
As an aside though it's still funny that the two tools WITH search also disagreed.
Earlier quoted context omitted.
Tried initially with a fifth bucket, Abstain. It was actually heavily used by some of the models. But it felt as if they are using this to "avoid" some of the hard questions, and we dropped this bucket to force them to provide a verdict.
> But it felt as if they are using this to "avoid" some of the hard questions, and we dropped this bucket to force them to provide a verdict. do you not see how that creates extremely misleading and valueless results? you are coercing the results into what you want to see.
Don't even need politics for it, there is no point in probing a mathematical black box for "how many soldiers died in the year X in war Y".
Any original source is preferable to a blurry "summary" of unknown sources, and this is why the article has a valuable point.
There's also no point in asking "Is Paris in France" either, if you substitute city and country with real data. An encyclopedia or manual check of different sources such as maps, while not infallible, is a better source.
If you already know the country Paris belongs to, there's no point in asking, anyway.