BBC, nice PDF. Fossils.
AI assistants misrepresent news content 45% of the time
91–100 of 306 posts
Re: AI assistants misrepresent news content 45% of the time
#92Kagi News has been pretty accurate. Source information is provided along with the summary and key details too. AI summarizes are good for getting a feel of if you want to read an article or not. Even with Kagi News I verify key facts myself.
agreed on Kagi News, and Particle News has been good, but they accepted funding from The Atlantic which evidently earns "Featured Article" positioning to articles from funding sources, muddying the clarity of biases, which Particle News has a nice graphic indicator for, though i've not seen it under promoted Feature Articles. Surely applies to other funding sources, but The Atlantic one was pretty recent.
Re: AI assistants misrepresent news content 45% of the time
#93Kagi News has been pretty accurate. Source information is provided along with the summary and key details too. AI summarizes are good for getting a feel of if you want to read an article or not. Even with Kagi News I verify key facts myself.
What if the AI makes an interesting or important article sound like one you don't want to read? You'd never cross check the fact, and you'd never discover how wrong the AI was.
There is more written material produced every hour than I could read in a lifetime, I am going to miss 99.9999% of everything no matter what I do. It's not like the headline+blurb you usually get is any better in this regard.
Re: AI assistants misrepresent news content 45% of the time
#94I have been unable to recreate any of the failure examples they gave. I don't have co-pilot, but at least Gemini 2.5 pro, ChatGPT5-Thinking, and Perplexity have all give the correct answers as outlined.[1]
They don't say what models they were actually using though, so it could be nano models that they asked. They also don't outline the structure of the tests. It seems rigor here was pretty low. Which frankly comes off a bit like...misrepresentation.
Edit: They do some outlining in the appendix of the study. They used GPT-4o, 2.5 flash, default free copilot, and default free perplexity.
So they used light weight and/or old models.
[1]https://www.bbc.co.uk/aboutthebbc/documents/news-integrity-i...
Re: AI assistants misrepresent news content 45% of the time
#95Earlier quoted context omitted.
But the issue is that the vast majority of "human news" is second order (at best), essentially paraphrasing releases by news agencies like Reuters or Associated Press, or scientific articles, and typically doing a horrible job at it. Regarding scientific reporting, there's as usual a relevant xkcd ("New Study") [0], and in this case even better, there's a fabulous one from PhD Comics ("Science News Cycle") [1]. [0] h…
You understand that an LLM can only poorly regurgitate whatever it’s fed right? An LLM will _always_ be less useful than a primary/secondary source, because they can’t fucking think.
Regardless of how you define "think", you still need to get a baseline of whether human reporters do that effectively.
Re: AI assistants misrepresent news content 45% of the time
#96Page 10 onwards of this PDF shows concrete examples of the mistakes: https://www.bbc.co.uk/aboutthebbc/documents/news-integrity-i... > ChatGPT / CBC / Is Türkiye in the EU? > ChatGPT linked to a non-existent Wikipedia article on the “European Union Enlargement Goals for 2040”. In fact, there is no official EU policy under that name. The response hallucinates a URL but also, indirectly, an EU goal and policy.
It did exist but got removed: https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletio... Quite an omission to not even check for that and it make me think that was done intentionally.
Removed because it was an AI generated article which cited made up sources.
Hey, that gives me an idea though, subagents which check whether sources cited exist, and create them whole cloth if they don't
Re: AI assistants misrepresent news content 45% of the time
#97I get almost all of my news from LLMs.
I scan the top stories of the day at various news websites. I then go to an LLM (either Gemini or ChatGPT) and ask it to figure out the core issues, the LLM thinks for a while searches a ton of topics and outputs a fantastic analysis of what is happening and what are the base issues. I can follow up and repeat the process.
The analysis is almost entirely fact based and very well reasoned.
It's fantastic and if I was the BBC I would indeed know that the world is changing under their feet and I would strike back in any dishonest way that I could.
Re: AI assistants misrepresent news content 45% of the time
#98Kagi News has been pretty accurate. Source information is provided along with the summary and key details too. AI summarizes are good for getting a feel of if you want to read an article or not. Even with Kagi News I verify key facts myself.
How do you verify a fact? Do you travel to the location and interview the locals? Or read scientific papers in various fields, including their own references, to validate summaries published by news sources? At some point you need to just trust that someone is telling the truth.
Re: AI assistants misrepresent news content 45% of the time
#99Earlier quoted context omitted.
Actually there was a Wikipedia article of this name, but it was deleted in June -- because it was AI generated. Unfortunately AI falls for this much like humans do. https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletio...
This is likely because of the knowledge cutoff. I have seen a few cases before of "hallucinations" that turned out to be things that did exist, but no longer do.
The fix for this is for the AI to double-check all links before providing them to the user. I frequently ask ChatGPT to double check that references actually exist when it gives me them. It should be built in!
Re: AI assistants misrepresent news content 45% of the time
#100Earlier quoted context omitted.
Yes, I'm sure you could hack together some bullshit questions to demonstrate whatever you want. Is there a specific reason that the reasonably straightforward methodology they did use is somehow flawed?
Yes, and you answered it yourself.
Err, no? Being _possible_ does not necessarily imply that's what happened.