Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assistants misrepresent news content 45% of the time

bbc.co.uk

Post reply on HN
Print view View original

201–210 of 306 posts

Re: AI assistants misrepresent news content 45% of the time

#201

Post by giantg2 »

latexr wrote: post #180
giantg2 wrote: post #143

"AI assistants misrepresent news content 45% of the time" How does that compare to the number for reporters? I feel like half the time I read or hear a report on a subject I know the reporter misrepresented something.

That’s whataboutism and doesn’t address the criticism or the problem. If a reporter misrepresents a subject, intentionally or accidentally, it doesn’t make it OK for a tool to then misrepresent it further, mangling both was correct and what was incorrect. https://en.wikipedia.org/wiki/Whataboutism

It's not whataboutism because I'm not using it to undermine the argument. It's a legitimate question to gauge the potential impact of an AI misrepresenting news. Assessing impact is part of determining corrective action and prioritization.

Re: AI assistants misrepresent news content 45% of the time

#203

Post by amarant »

scarmig wrote: post #38

If you dig into the actual report (I know, I know, how passe), you see how they get the numbers. Most of the errors are "sourcing issues": the AI assistant doesn't cite a claim, or it (shocking) cites Wikipedia instead of the BBC. Other issues: the report doesn't even say which particular models it's querying [ETA: discovered they do list this in an appendix], aside from saying it's the consumer tier. And it leaves o…

Human journalists misrepresent the white paper 85% of the time.

With this in mind, 45% doesn't seem so bad anymore

Re: AI assistants misrepresent news content 45% of the time

#204

Post by extrabajs »

menaerus wrote: post #124
darkwater wrote: post #100

Earlier quoted context omitted.

Err, no? Being _possible_ does not necessarily imply that's what happened.

A bucket of 30 questions is not a statistically significant sample size which we can use to support the hypothesis which goes to say that all AI assistants they tested are 45% of the time wrong. That's not how science works. Neither is my bucket of 30 questions statistcally significant but it goes to say that I can disprove their hypothesis just by giving them my sample. I think that the report is being disingenious…

Statistically significant... sample size? Support the hypothesis?

Re: AI assistants misrepresent news content 45% of the time

#205

Post by dan_h »

alcide wrote: post #12

Kagi News has been pretty accurate. Source information is provided along with the summary and key details too. AI summarizes are good for getting a feel of if you want to read an article or not. Even with Kagi News I verify key facts myself.

I've has a similar experience with my own project that summarizes rss articles--the results have largely been pretty good, but I found using a "reasoning" model had much better results.

Re: AI assistants misrepresent news content 45% of the time

#206

Post by dns_snek »

Narciss wrote: post #63

> All participating organizations then generated responses to each question from each of the four AI assistants. This time, we used the free/consumer versions of ChatGPT, Copilot, Perplexity and Gemini. Free versions were chosen to replicate the default (and likely most common) experience for users. Responses were generated in late May and early June 2025. First of all, none of the SOTA models we're currently using w…

Ah, the "you're using the wrong model" fallacy (is there a name for this?)

In the eyes of the evangelists, every major model seems to go from "This model is close to flawless at this task, you MUST try this TODAY" to "It's absolutely wild that anyone would ever consider using such a no-good, worthless model for this task" over the course of a year or so. The old model has to be re-framed for the new model to look more impressive.

When GPT-4 was released I was told it was basically a senior-level developer, now it's an obviously worthless model that you'd be a fool to use to write so much as a throwaway script.

Re: AI assistants misrepresent news content 45% of the time

#207

Post by mdhb »

rideontime wrote: post #105
1980phipsi wrote: post #99

Earlier quoted context omitted.

The fix for this is for the AI to double-check all links before providing them to the user. I frequently ask ChatGPT to double check that references actually exist when it gives me them. It should be built in!

But that would mean OpenAI would lose even more money on every query.

Almost as though it’s not a sustainable business model and relies of tricking people in order to keep the lights on.

Re: AI assistants misrepresent news content 45% of the time

#208

Post by visarga »

mckngbrd wrote: post #183
visarga wrote: post #20

I recently tried to get Gemini to collect fresh news and show them to me, and instead of using search it hallucinated everything wholesale, titles, abstracts and links. Not just once, multiple times. I am kind of afraid of using Gemini now for anything related to web search. Here is a sample: > [1] Google DeepMind and Harvard researchers propose a new method for testing the ‘theory of mind’ of LLMs - Researchers have…

What version of Gemini were you using? i.e. were you calling it locally via the API or thru their Gemini or AI Studio web apps? Not every LLM app has access to web / news search capabilities turned on by default. This makes a huge difference in what kind of results you should expect. Of course, the AI should be aware that it doesn't have access to web / news search, and it should tell you as much rather than hallucin…

Gemini 2.5 Pro and it was this month, so probably the latest version.

Re: AI assistants misrepresent news content 45% of the time

#209

Post by xpe »

TL;DR: I recommend downloading and reading the "News Integrity in AI Assistants TOOLKIT" (PDF) [1] linked from the article.

=Why?= The PDF is something that can appeal to anyone who is simply striving to have slower, deeper conversations about AI and the news.

=Frustration= No matter where you land on AI, it seems to me most of us are tired of various framings and exaggerations in the news. Not the same ones, because we often disagree! We feel divided.

=The Toolkit= The European Broadcasting Union (EBU) and BBC have laid out their criteria in this report "News Integrity in AI Assistants Toolkit" [1] IMO, it is the hidden gem from the whole article.

- Let me get the obvious flaws out of the way. (1) Yes, it is a PDF. (2) It is nothing like a software toolkit. (3) It uses the word taxonomy, which conjures brittle and arbitrary tree classification systems -- or worse, the unspeakable horror of ontology and the lurking apparently-unkillable hydra that is the Semantic Web.

- But there are advantages too. With a PDF, you can read it without ads or endless scrolling. This PDF is clear. It probably won't get you riled up in a useless way. It might even give you some ideas of what you can do to improve your own news consumption or make better products.

All in all, this is a PDF I would share with almost anyone (who reads English). I like that it is dry, detailed, and, yes a little boring.

[1]: https://www.bbc.co.uk/aboutthebbc/documents/news-integrity-i...

Re: AI assistants misrepresent news content 45% of the time

#210

Post by visarga »

HWR_14 wrote: post #114
visarga wrote: post #20

I recently tried to get Gemini to collect fresh news and show them to me, and instead of using search it hallucinated everything wholesale, titles, abstracts and links. Not just once, multiple times. I am kind of afraid of using Gemini now for anything related to web search. Here is a sample: > [1] Google DeepMind and Harvard researchers propose a new method for testing the ‘theory of mind’ of LLMs - Researchers have…

Why would you want Gemini to do this instead of just going to a news site (or several news sites) and reading what the headlines they wrote?

I wanted to use the agentic powers of the model to dig for specific kinds of news, and use iterative search as well. I think when LLMs use tools correctly this kind of search is more powerful than simple web search. It also has better semantic capabilities, so in a way I wanted to make my own LLM powered news feed.
Post reply on HN