Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assistants misrepresent news content 45% of the time

bbc.co.uk

Post reply on HN
Print view View original

161–170 of 306 posts

Re: AI assistants misrepresent news content 45% of the time

#161

Post by biophysboy »

BeetleB wrote: post #85
biophysboy wrote: post #80

Earlier quoted context omitted.

How is that possible if the AI models rely on and implicitly trust these sources?

The article is about how well AI models misrepresent the content of news , not how often they misrepresent reality. My point is that even if the AI models make no errors when representing news content, they'll still be quite inaccurate when reality is the benchmark. Who cares if AI does a good job representing the source, when the source is crap?

Yes, if AI represents news, and news tries (and often fails) to represent reality, then AI would represent reality 0.55*0.55 of the time, taking both your claim and BBC's claim as true. That is even worse than the already low bar for news you and I agree on.

Re: AI assistants misrepresent news content 45% of the time

#162

Post by FooBarWidget »

542458 wrote: post #129
FooBarWidget wrote: post #126

Earlier quoted context omitted.

I wouldn't even say BBC is a good source to cite. For foreign news, BBC is outright biased. Though I don't have any good suggestions for what an LLM should cite instead.

Reuters or AP IMO. Both take NPOV and accuracy very seriously. Reuters famously wouldn't even refer to the 9/11 hijackers as terrorists, as they wanted to remain as value-neutral as possible.

It's been a long time since 2001. Are they still value-neutral today on foreign news? It seems to me like they're heavily biased towards western POV nowadays.

Re: AI assistants misrepresent news content 45% of the time

#163

Post by fullofideas »

gamerDude wrote: post #117
bunderbunder wrote: post #108

Earlier quoted context omitted.

The biggest problem with that citation isn't that the article has since been deleted. The biggest problem is that that particular Wikipedia article was never a good source in the first place. That seems to be the real challenge with AI for this use case. It has no real critical thinking skills, so it's not really competent to choose reliable sources. So instead we're lowering the bar to just asking that the sources a…

I think this is a real challenge for everyone. In many ways potentially we need a restart of a wikipedia like site to document all the valid and good sources. This would also hopefully include things like source bias and whether it's a primary/secondary/tertiary source.

This is pushing the burden of proof on the society. Basically, asking everyone else to pitch in and improve sources so that ai companies can reference these trust worthy sources.

Re: AI assistants misrepresent news content 45% of the time

#164

Post by dontlaugh »

gadders wrote: post #133
dontlaugh wrote: post #130

Earlier quoted context omitted.

The BBC has a strong right wing bias within the UK too. There’s no such thing as unbiased.

[flagged]

The BBC is famous for platforming Farage and smearing Corbyn.

The Guardian is at best centre-right.

Next you’ll try to convince me that Starmer’s Labour is left wing or the Lock Ness monster is real.

Re: AI assistants misrepresent news content 45% of the time

#166

Post by lyu07282 »

smt88 wrote: post #144
senordevnyc wrote: post #110

Earlier quoted context omitted.

I'm curious if LLM skeptics bother to click through and read the details on a study like this, or if they just reflexively upvote it because it confirms their priors. This is a hit piece by a media brand that's either feeling threatened or is just incompetent. Or both.

Whether a hitpiece or not, it rhymes with my experience and provides receipts. Can you provide yours?

Because yours is anecdotal evidence, a study like this should have a higher bar than that and be useful to support your experience, but it doesn't do that. It doesn't even say what exact models they evaluated ffs

Re: AI assistants misrepresent news content 45% of the time

#167

Post by zahlman »

janwl wrote: post #125
1980phipsi wrote: post #99

Earlier quoted context omitted.

The fix for this is for the AI to double-check all links before providing them to the user. I frequently ask ChatGPT to double check that references actually exist when it gives me them. It should be built in!

I thought people here hated it when LLMs made http requests?

It's bad when they indiscriminately crawl for training, and not ideal (but understandable) to use the Internet to communicate with them (and having online accounts associated with that etc.) rather than running them locally.

It's not bad when they use the Internet at generation time to verify the output.

Re: AI assistants misrepresent news content 45% of the time

#168

Post by vidarh »

n4r9 wrote: post #151
vidarh wrote: post #33

Earlier quoted context omitted.

> Now let's run this experiment against the editorial boards in newsrooms. Or against people in general. It's a pet peeve of mine that we get these kinds of articles without a baseline established of how people do on the same measure. Is misrepresenting news content 45% of the time better or worse than the average person? I don't know. By extension: Would a person using an AI assistant misrepresent news more or less…

The difference is the ease with which AI can be rolled out, scaled up, and woven into the fabric of our interactions with society.

That makes understanding the baseline all the more important. It could be a disaster, or it could in fact be a distinct improvement. Every time someone pushes a breathless headline about failure rates of AI without comparing it to a human baseline, they are in essence potentially misleading us because without that baseline we don't know whether it's better or worse.

Re: AI assistants misrepresent news content 45% of the time

#170

Post by magackame »

simonw wrote: post #146
brabel wrote: post #89

Earlier quoted context omitted.

It did exist but got removed: https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletio... Quite an omission to not even check for that and it make me think that was done intentionally.

It's probably for the best that chat interfaces avoid making direct HTTP calls to sources at run-time to confirm that they don't 404 - imagine how much extra traffic that could add to an internet ecosystem which is suffering from badly written crawlers already. (Not to mention plenty of sites have added robots.txt rules deliberately excluding known AI user-agents now.)

Wouldn't it be the same amount of requests as a regular person researching something the old way?
Post reply on HN