Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

131–140 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#131

Post by johnea »

glial wrote: post #106
johnea wrote: post #5

> Internally, it uses a sophisticated, multi-path strategy, approximating the sum with one heuristic while precisely determining the final digit with another. Yet, if asked to explain its calculation, the LLM describes the standard 'carry the one' algorithm taught to humans. So, the LLM isn't just wrong, it also lies...

Talking about "truth" or "lies" with LLMs isn't helpful.

Could you get the CEO of Goggle or OpenAI to state that clearly in a press announcement? 8-)

Although "isn't helpful" is rather dodgy wording. "Helpful" for who? "Helpful" in what way?

I think most users would find it helpful if the output was not presented as correct, when it's incorrect.

If every prompt that ever went into an LLM was prefixed with: "tell me a made up story about:", then the user expectation would be more in line with what the output represents.

But, that's not the way the corps are describing it, is it?

Re: The Unreliability of LLMs and What Lies Ahead

#132

Post by worldsayshi »

brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

This doesn't seem like the first time engineers try to work with something useful that is only partially reliable.

The solution is to play at its strengths and reinforce it with other mediums. You don't build structures with pure concrete. You add rebar. You don't build ships out of only sail and you don't build rail with just iron. You compose materials in a way that makes sense.

LLMs are most useful when the output is immediately verifiable. So let's build frameworks that take that to core. Build everything around verification. And use LLMs for its strengths.

Re: The Unreliability of LLMs and What Lies Ahead

#133

Post by johnea »

GuB-42 wrote: post #73
johnea wrote: post #5

> Internally, it uses a sophisticated, multi-path strategy, approximating the sum with one heuristic while precisely determining the final digit with another. Yet, if asked to explain its calculation, the LLM describes the standard 'carry the one' algorithm taught to humans. So, the LLM isn't just wrong, it also lies...

A LLM can't self-reflect. It doesn't know what happens in its own circuits. If you ask it, it will either tell you what it knows (from the articles about LLMs it has ingested), and if it doesn't, it will hallucinate something, as it is often the case. Since the LLM has no knowledge on how LLMs do addition, it will pick something that seems to makes sense, and it picked the "carry the one" algorithm. New generations o…

Please see the reply to the comment above...

Re: The Unreliability of LLMs and What Lies Ahead

#134

Post by eterm »

netsharc wrote: post #126
eterm wrote: post #120

Earlier quoted context omitted.

We're getting into semantics now, but I'm talking about the kind of person who used to sit in a physical store, waiting for someone to walk by and go into the travel agency. In the 80's and 90's, this is how most people booked their holidays. It was labour intensive, people would spend some time talking with a travel agent in a store, who would have a good idea of the packages available, and be able to make recommend…

I imagine a travel agent would have local knowledge and connections, and would know the quality of the hotels they're trying to send to you, a high commission isn't worth it if your customer is unsatisfied and goes to a different agent for their next trip. Of course this is based on the assumption that the customer always wants to use a travel agent (an unrealistic assumption nowadays, because it's so easy to switch…

What you're imagining is what it was like in the 1980s, or possibly now for a boutique place, not the reality of the post-internet high-street travel agent.

You're not realising the reality of the typical high-street worker, and the sheer lack of autonomy that they have in their roles.

Re: The Unreliability of LLMs and What Lies Ahead

#135

Post by insane_dreamer »

andrewmutz wrote: post #98
brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

What we are seeing with our customers is that LLM errors are a very manageable problem. End users adapt pretty quickly to the idea that AI systems aren't perfect. In many cases AI products are doing tasks that used to be done by humans and these humans were making mistakes too, so the end user is used to the idea that the task will get accomplished with some non-zero error rate. You just need to build your products i…

> the user has the ability to easily double check the results whenever they like

if the user is able to so easily verify that the results are accurate, that means that they are able to generate accurate results through other means, which means they don't need the LLM in the first place

Re: The Unreliability of LLMs and What Lies Ahead

#136

Post by yencabulator »

bagacrap wrote: post #113
tveita wrote: post #110

Earlier quoted context omitted.

> If a calculator works great 99% of the time you could not use that calculator to build a bridge. We know for certain that certified lawyers have committed malpractice by using ChatGPT, in part because the made-up citations are relatively easy to spot. Malpractice by engineers might take a little more time to discover.

Engineers' work is also externally verifiable, e.g. by unit tests for software, but I'm assuming by other sorts of automated protocols for civil engineering. I would hope a bridge is not built without triple checking the various outcomes.

If that argument were to save anyone, it would have saved the lawyers too.

Re: The Unreliability of LLMs and What Lies Ahead

#137

Post by yencabulator »

Spivak wrote: post #78
ok123456 wrote: post #64

MongoDB was basically "vibe coding" for RBDMs. After the hype cycle, there will be a wasteland of unmaintainable vibe-coded products that companies will have to pump unlimited amounts of money into to maintain.

I think we mythologize the relational model a bit too much to call nosql dbs vibe coding. DynamoDB is quite good and you can point to some very large customers using it successfully.

MongoDB was bad for several reasons unrelated to the relational model.

Re: The Unreliability of LLMs and What Lies Ahead

#138

Post by orwin »

bagacrap wrote: post #113
tveita wrote: post #110

Earlier quoted context omitted.

> If a calculator works great 99% of the time you could not use that calculator to build a bridge. We know for certain that certified lawyers have committed malpractice by using ChatGPT, in part because the made-up citations are relatively easy to spot. Malpractice by engineers might take a little more time to discover.

Engineers' work is also externally verifiable, e.g. by unit tests for software, but I'm assuming by other sorts of automated protocols for civil engineering. I would hope a bridge is not built without triple checking the various outcomes.

Well, most of the LLM-generated code i serve are unit tests (and scripts), so hopefully, those are good enough to catch my mistakes :)

Re: The Unreliability of LLMs and What Lies Ahead

#139

Post by jimbokun »

asadotzler wrote: post #82
eterm wrote: post #44

There are jobs out there that have always been unreliable. A classic example is the Travel Agent. This was already a job driven to near-extinction just by Google, but LLMs are a nail in the travel agent coffin. The job was always fuzzy. It was always unreliable. A travel agent recommendation was never a stamp of quality or guarentee of satisfaction. But now, I can ask an LLM to compare and contrast two weeks in the S…

If you don't care about reliability, repeatability and accuracy, they're great.

This should be OpenAI's official slogan!

Re: The Unreliability of LLMs and What Lies Ahead

#140

Post by jimbokun »

whyowhy3484939 wrote: post #77
eterm wrote: post #44

There are jobs out there that have always been unreliable. A classic example is the Travel Agent. This was already a job driven to near-extinction just by Google, but LLMs are a nail in the travel agent coffin. The job was always fuzzy. It was always unreliable. A travel agent recommendation was never a stamp of quality or guarentee of satisfaction. But now, I can ask an LLM to compare and contrast two weeks in the S…

Once they start making deals with the relevant organizations, book rooms, handle insurance, replacement hotels, etc, then they'll replace travel agents. These guys don't just Google a bunch of tickets you know.

Um, Google and travel sites already replaced travel agents a LONG time ago.
Post reply on HN