Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assistants misrepresent news content 45% of the time

bbc.co.uk

Post reply on HN
Print view View original

71–80 of 306 posts

Re: AI assistants misrepresent news content 45% of the time

#71

Post by JumpCrisscross »

vidarh wrote: post #33
MangoToupe wrote: post #5

Now let's run this experiment against the editorial boards in newsrooms. Obviously, AI isn't an improvement, but people who blindly trust the news have always been credulous rubes. It's just that the alternative is being completely ignorant of the worldviews of everyone around you. Peer-reviewed science is as close as we can get to good consensus and there's a lot of reasons this doesn't work for reporting.

> Now let's run this experiment against the editorial boards in newsrooms. Or against people in general. It's a pet peeve of mine that we get these kinds of articles without a baseline established of how people do on the same measure. Is misrepresenting news content 45% of the time better or worse than the average person? I don't know. By extension: Would a person using an AI assistant misrepresent news more or less…

> It's a pet peeve of mine that we get these kinds of articles without a baseline established of how people do on the same measure

I don’t have a personal human news summarizer?

The comparison is between a human reading the primary source against the same human reading an LLM hallucination mixed with an LLM referring the primary source.

> cynic in me want another question answered too: How often does reporters misrepresent the news?

The fact that you mark as cynical a question answered pretty reliably for most countries sort of tanks the point.

Re: AI assistants misrepresent news content 45% of the time

#72

Post by incomingpain »

http://www.aaronsw.com/weblog/hatethenews

I've been thinking about the state of our media, and the crisis of trust in news began long before AI.

We have a huge issue, and the problem is with the producers and the platform.

I'm not talking about professional journalists who make an honest mistake, own up to it with a retraction, and apologize. I’m talking about something far more damaging: the rise of false journalists, who are partisan political activists whose primary goal is to push a deliberately misleading or false narrative.

We often hear the classic remedy for bad speech: more speech, not censorship. The idea is that good arguments will naturally defeat bad ones in the marketplace of ideas.

Here's the trap: these provocateurs create content that is so outrageously or demonstrably false that it generates massive engagement. People are trying to fix their bad speech with more speech. And the algorithm mistakes this chaotic engagement for value.

As a result, the algorithm pushes the train wreck to the forefront. The genuinely good journalists get drowned out. They are ignored by the algorithm because measured, factual reporting simply doesn't generate the same volatile reaction.

The false journalists, meanwhile, see their soaring popularity and assume it's because their "point" is correct and it's those 'evil nazis from the far right who are wrong'. In reality, they're not popular because they're insightful; they're popular because they're a train wreck. We're all rubbernecking at the disaster and the system is rewarding them for crashing the integrity of our information.

Re: AI assistants misrepresent news content 45% of the time

#73

Post by biophysboy »

Narciss wrote: post #63

> All participating organizations then generated responses to each question from each of the four AI assistants. This time, we used the free/consumer versions of ChatGPT, Copilot, Perplexity and Gemini. Free versions were chosen to replicate the default (and likely most common) experience for users. Responses were generated in late May and early June 2025. First of all, none of the SOTA models we're currently using w…

If they used a paid version, their study would not represent how most people use AI (with the free version)

Re: AI assistants misrepresent news content 45% of the time

#75

Post by underlipton »

Workaccount2 wrote: post #59

The media today is so polarized, so dishonest, and so bent on feeding the egos of it's users, the bar to pass them is literally underground. You can go through most big name media stories and find it ridden with omissions of uncomfortable facts, careful structuring of words to give the illusion of untrue facts being true, and careful curation of what stories are reported. More than anything, I hope AI topples the gar…

All of this is true, and LLMs' nature as stochastic parrots mean that they'll do pretty much nothing to stem the tide. Journalism needs to be somewhere between the USPS, USAID, and local school boards: a network of local and independent offices, funded mostly by guaranteed government grants, reporting judiciously and independently of how the content squares with any particular group's interests. And if anyone wants to curate that feed, fine, but the feed would be there for all to peruse.

Re: AI assistants misrepresent news content 45% of the time

#76

Post by nopinsight »

Hallucination Leaderboard "This evaluates how often an LLM introduces hallucinations when summarizing a document."

https://github.com/vectara/hallucination-leaderboard

If the figures on this leaderboard are to be trusted, many frontier and near-frontier models are already better than the median white-collar worker in this aspect.

Note: The leaderboard doesn't cover tool calling, to be clear.

Re: AI assistants misrepresent news content 45% of the time

#77

Post by roguecoder »

I am curious if LLMs evangelists understand how off-putting it is when they knee-jerk rationalize how badly these tools are performing. It makes it seem like it isn't about technological capabilities: it is about a religious belief that "competence" is too much to ask of either them or their software tools.

Re: AI assistants misrepresent news content 45% of the time

#79

Post by Yizahi »

visarga wrote: post #20

I recently tried to get Gemini to collect fresh news and show them to me, and instead of using search it hallucinated everything wholesale, titles, abstracts and links. Not just once, multiple times. I am kind of afraid of using Gemini now for anything related to web search. Here is a sample: > [1] Google DeepMind and Harvard researchers propose a new method for testing the ‘theory of mind’ of LLMs - Researchers have…

But LLM can't collect anything. It can generate the most likely characters in a row. What exactly did you expect from it?

Re: AI assistants misrepresent news content 45% of the time

#80

Post by biophysboy »

BeetleB wrote: post #74

Actual news articles misrepresent reality more often than 45%. Some very recent discussions on HN: https://news.ycombinator.com/item?id=45617088 https://news.ycombinator.com/item?id=45585323

How is that possible if the AI models rely on and implicitly trust these sources?
Post reply on HN