Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

151–160 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#151

Post by 112233 »

BlueTemplar wrote: post #142
ar813 wrote: post #59

If I take a step back and think back to say a few (or 5) years ago, what LLMs can do is amazing. One has to acknowledge that (or at least, I do). But as a scientist it's been rather interesting to probe the jagged edge and unreliability, including using deep research tools, on any topic I know well. If I read through the reports and summaries it generates, it seems at first glance correct - the jargon is used correct…

You know who else is infamous for making errors due to shallow understanding ? (Non-specialized) journalists ! How do you find they compare?

Not OP, but here is my observations: The llm are uniformly dumb and not "understaging" across all spectrum of topics. It is counter-intuitive. By asking llm to simply blab ("write a story about ..") you notice it:

- mixes up pronouns (who is "you" or "he")

- cannot keep track of what is where.

- continuously plugs it's guidance slant ("lets cook dinner, Bob! It is paramount to strive for safety and cooperation while doing it!")

— language style is all over the place, comically so.

— when asked about the text it just generated, is able to give valid critique to itself (i.e. having that "insight" does not help the generation)

Journalists may have shallow understanding of topic, but they do not start referring to a person they write about as "me" halfway through.

LLM is uniformly dumb

Re: The Unreliability of LLMs and What Lies Ahead

#152

Post by helloplanets »

mjburgess wrote: post #128
helloplanets wrote: post #125

Earlier quoted context omitted.

You need to be pushing much more data in than you're getting out. 40k tokens of input can result in 400 actual quality tokens of output. Not giving enough input to work off of will result in regressed output. It's basically like a funnel, which can also be used the other way around if the user is okay with quirky side effects. It feels like a lot of people are using the funnel the wrong way around and complaining tha…

Sure, if you have a high-quality starting point and need refinement. The issue is that the vast majority of user-facing LLM use cases are where people don't have these high-quality starting points. They don't have 40k tokens to make 400.

You can just attach 40k of context directly into the Gemini, ChatGPT and Claude web interfaces afaik. If someone is using an LLM as a tool to actually be of help in an area they are already professionals in, conjuring good books, research, etc, as attachments shouldn't be an issue.

But yes, the default mode of LLMs is usually a WikiHow and content farm style answer. This is also a problem with Google: The content you get back from generic searches will often be riddled with inaccuracies and massive generalizations.

Not being able / bothering to come up with relevant context and throwing the dice on the LLM being able to do this out of the box is definitely a serious issue. I really think that is where the discussion should be: Focused more on how people use these tools. Just like you can tell quite a bit about someone's expertise based on the specific way in which they interface with Google (or any information on the internet) while they work.

Re: The Unreliability of LLMs and What Lies Ahead

#153

Post by psychoslave »

johnea wrote: post #5

> Internally, it uses a sophisticated, multi-path strategy, approximating the sum with one heuristic while precisely determining the final digit with another. Yet, if asked to explain its calculation, the LLM describes the standard 'carry the one' algorithm taught to humans. So, the LLM isn't just wrong, it also lies...

No, because the LLM is a tool without any feeling and consciousness, like the article rightfully point out. It doesn't have the possibility to scrutinize it's own internals, nor the possibility to wonder if that would be something relevant to do.

Those who lie (possibly even to themselves) are those who pretend that mimicry if stretched enough will surpass the actual thing, and foster the deceptive psychological analogies like "hallucinate".

Re: The Unreliability of LLMs and What Lies Ahead

#155

Post by franktankbank »

oconnor663 wrote: post #146
brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

A pedantic but maybe-not-entirely-pedantic point: It depends on what you mean by 99%. If the calculator has a little gremlin in it that rolls a random 100-sided die, and gives you the wrong answer every time it rolls a 1, then you certainly can use it to build a bridge. You just need to do each calculation say 10 or 20 times and take the majority answer :) If the gremlin is clever, it might remember the wrong answers…

Yea I know, can't really understand why people have such a problem with this. Just ignore the wrong answers and be thankful when it gives you a right answer. Picky bastards.

Re: The Unreliability of LLMs and What Lies Ahead

#156

Post by Ukv »

insane_dreamer wrote: post #135
andrewmutz wrote: post #98

Earlier quoted context omitted.

What we are seeing with our customers is that LLM errors are a very manageable problem. End users adapt pretty quickly to the idea that AI systems aren't perfect. In many cases AI products are doing tasks that used to be done by humans and these humans were making mistakes too, so the end user is used to the idea that the task will get accomplished with some non-zero error rate. You just need to build your products i…

> the user has the ability to easily double check the results whenever they like if the user is able to so easily verify that the results are accurate, that means that they are able to generate accurate results through other means, which means they don't need the LLM in the first place

I don't think that's necessarily true - many tasks are difficult to solve but easy to verify. If I ask "place names that end with um", or "good ideas for a birthday party" I can pretty much verify the answer just by reading it. In other cases, clicking through to check that a linked source supports a claim is easier than researching to find and summarize the source in the first place would be.

Re: The Unreliability of LLMs and What Lies Ahead

#157

Post by willk357 »

Has anyone experimented with an ensemble + synthesizer approach for reliability? I'm thinking: make n identical requests to get diverse outputs, then use a separate LLM call to synthesize/reconcile the distinct results into a final answer. Seems like it could help with the consistency issues discussed here by leveraging the natural variance in LLM outputs rather than fighting it. Any experience with this pattern?

Re: The Unreliability of LLMs and What Lies Ahead

#158

Post by banannaise »

vinni2 wrote: post #117
brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

> If a calculator works great 99% of the time you could not use that calculator to build a bridge. But if the alternative is doing calculations by hand (writing code manually) there is a higher chance of making mistakes. Just like calculations are double checked while building bridges unit tests and code reviews should catch bugs introduced by LLM written code.

Code review is your last (and worst) line of defense. Humans are not good at needle-in-a-haystack tasks.

Re: The Unreliability of LLMs and What Lies Ahead

#159

Post by insane_dreamer »

Ukv wrote: post #156
insane_dreamer wrote: post #135

Earlier quoted context omitted.

> the user has the ability to easily double check the results whenever they like if the user is able to so easily verify that the results are accurate, that means that they are able to generate accurate results through other means, which means they don't need the LLM in the first place

I don't think that's necessarily true - many tasks are difficult to solve but easy to verify. If I ask "place names that end with um", or "good ideas for a birthday party" I can pretty much verify the answer just by reading it. In other cases, clicking through to check that a linked source supports a claim is easier than researching to find and summarize the source in the first place would be.

> I can pretty much verify the answer just by reading it

Only if you have domain knowledge. In both of your examples, you have to 1) know geography to determine whether "Técolum" and "Tolum" are indeed city names or just made up; and 2) know what might be acceptable ("good idea") or not at a birthday party.

Yes, it'll probably save you some time, but it's not orders of magnitude.

> In other cases, clicking through to check that a linked source supports a claim

this supposes that the AI provides a link for every fact. Google search + Gemini does, but most LLM interfaces don't.

secondly, if I have to click through every link and read through the source to determine whether details of a "summary" are correct or not, that really does not save me much time from conducting a search and looking through the linked sources myself

Anecdote from a couple of weeks ago. My wife's professor sent her 5 citations and summaries related to a medical research project. She didn't say they were LLM generated, but it was obvious (to me, not my wife) they were, by the formatting alone. None of the 5 papers existed as cited. My wife was confused, spent a lot of time trying to figure out what was wrong and why she couldn't find any of the papers. A Google Scholar search turned up 2 of the papers which were close enough to the citation to be the ones with some logical thinking, but the other 3 were not even matchable. In the end, the time spent trying to sort out valid vs invalid citations, and find valid replacements, was significantly greater than just doing the search and looking through the abstracts.

PS: LLMs are fine for information that can be "fuzzy": suggest places to go on vacation in September, plan a birthday party, etc. But I wouldn't consider that to be a "revolutionary" advance.

Re: The Unreliability of LLMs and What Lies Ahead

#160

Post by johnea »

psychoslave wrote: post #153
johnea wrote: post #5

> Internally, it uses a sophisticated, multi-path strategy, approximating the sum with one heuristic while precisely determining the final digit with another. Yet, if asked to explain its calculation, the LLM describes the standard 'carry the one' algorithm taught to humans. So, the LLM isn't just wrong, it also lies...

No, because the LLM is a tool without any feeling and consciousness, like the article rightfully point out. It doesn't have the possibility to scrutinize it's own internals, nor the possibility to wonder if that would be something relevant to do. Those who lie (possibly even to themselves) are those who pretend that mimicry if stretched enough will surpass the actual thing, and foster the deceptive psychological anal…

The LLM doesn't have a brain, it doesn't have consciousness, therefore it doesn't "hallucinate"; it just produces factually incorrect results.

It's just wrong, and then gives misleading explanations of how it got the wrong answer, following the same process that led to the wrong answer in the first place. Lying is a subset of being wrong.

The tech has great applications, why hype the stuff it doesn't do well? Or apply terms that misrepresent the process the s/w uses?

One might say the use of the word "hallucinate" is an analogy, but it's a poor analogy, which further misleads the lay public in what is actually happening inside the LLM, and how it's results are generated.

If you want to assert that "hallucinate" is an analogy, then "lying" is also an analogy.

If every prompt that ever went into an LLM was prefixed with: "Tell me a made up story about: ...", then the user expectation would be more in line with what the output represents.

I'm not averse to the tech in general, but I am against the rampant misrepresentation that's going on...

Post reply on HN