The Unreliability of LLMs and What Lies Ahead • Hacker News Board

Re: The Unreliability of LLMs and What Lies Ahead

#21

Post by dingnuts » Mon, Jun 02, 2025, 7:33 PM UTC

kenjackson wrote: post #14

nomel wrote: post #9

Earlier quoted context omitted.

From what I can tell, rather than a simple difference in expectation (which could explain your positive experience vs others), it seems to be a "comfort within uncertainty" difference that, from what I can tell, is a personality trait! You're comfortable with the uncertainty, and accommodate it in your use and expectations. You're left feeling good about the experience, within that uncertainty. Others are repelled by…

I wonder if this is like dishwasher usage. As a kid growing up we never used the dishwasher. It was just the drying rack. The reason was you had to rinse off the big stuff anyways, and then the resulting quality of dishwashing was poor in it. You'd often get a fork with rice stuck between it still, which was unacceptable. As a grown up now I use a dishwasher for everything that is permitted to go in it. I still have…

This is a funny example since, for a long time anyway, dishwashers have been much better at actually sanitizing dishes due to the much higher temperatures that can be used vs hand washing. I don't feel like hand washed dishes are truly clean. Oh you rubbed it with a nasty dish rag and water cool enough to touch? greeeeaaaaat

Re: The Unreliability of LLMs and What Lies Ahead

#22

Post by gte525u » Mon, Jun 02, 2025, 7:33 PM UTC

leptons wrote: post #10

jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

It's fine if LLMs are used casually, for things that don't affect anyone but the user. But when someone plugs an LLM into Social Security or other governmental bodies to take action on real human beings, then disaster awaits. Nobody is going to care if the LLM got it wrong if you're just chatting with it or writing some wonky code that doesn't matter in the real world, but when your government check is reduced or del…

^This - we're trying to use one to partially automate some system engineering type activities.

It's great for reviews where any given reviewer could be expected to have a misunderstanding of certain details or skip a section (RAG somewhat helps this) - but it's frustrating for artifact generation where missing details cascade through the project.

As great as the technology (right now) it seems so far from reliable business process automation.

Re: The Unreliability of LLMs and What Lies Ahead

#23

Post by AlienRobot » Mon, Jun 02, 2025, 7:35 PM UTC

I'm no AI fan, but articles talking about the shortcomings of LLM's seem to have to be complaining that forks aren't good for drinking soup.

Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry.

For the love of God. It's not actual intelligence. This isn't hard. It just randomly spits out text. Use it for what it's good at instead. Text.

Instead of hunting for how to do things in programming using an increasingly terrible search engine, I just ask ChatGPT. For example, this is something I've asked ChatGPT in the past:

    in typescript, I have a type called IProperty, how do I create a function argument that receives a tuple of IProperty of various T types and returns a tuple of the T types of the IProperty in order received?

This question that's such an edge case that I wasn't even sure how to word properly actually yielded the answer I was looking for.

    function extractValues[]>(
      props: [...T]
    ): { [K in keyof T]: T[K] extends IProperty ? U : never } {
      return props.map(p => p.get()) as any;
    }

This doesn't look unrealiable to me. It actually feels pretty useful. I just need [...T] there and infer there.

Re: The Unreliability of LLMs and What Lies Ahead

#24

Post by keybrd-intrrpt » Mon, Jun 02, 2025, 7:35 PM UTC

bluefirebrand wrote: post #19

kenjackson wrote: post #14

Earlier quoted context omitted.

I wonder if this is like dishwasher usage. As a kid growing up we never used the dishwasher. It was just the drying rack. The reason was you had to rinse off the big stuff anyways, and then the resulting quality of dishwashing was poor in it. You'd often get a fork with rice stuck between it still, which was unacceptable. As a grown up now I use a dishwasher for everything that is permitted to go in it. I still have…

Imagine if the advice for Dishwasher usage mirrored the advice for AI "You have to iterate on the output to get good results" Just keep running that dishwasher until they're clean! If you run it and they're still dirty, load it up with soap and try again!

That's all new technology though. Dishwashers _were_ like that.

What's seemed to change are people's expectations of technology that "just works". When in reality, we are in the infant years of AI/ML and LLMs

We're so spoiled by the pace of innovation we're upset it requires a bit of hand-holding while they figure things out.

Re: The Unreliability of LLMs and What Lies Ahead

#25

Post by keybrd-intrrpt » Mon, Jun 02, 2025, 7:38 PM UTC

dist-epoch wrote: post #4

josefritzishere wrote: post #2

It's hard to say "never" in technology. History isn't really on your side. However, LLMs have largely proven to be good at things computers were are already good at: repetitive tasks, parallel processing, and data analysis. There's nothing magical about an LLM that seems to be defeating the traditional paradigm. Increasingly I lean toward an implosion of the hype cycle for AI.

Funny, I don't remember any computer program in the past being able to explain a news article through the lens of one particular philosopher. Or being able to explain the static physical forces in a picture that are keeping a structure from collapsing. Or recommend me a python library which does X, Y and Z with constraints A, B and C. But I guess you can file all the above under "data analysis".

It's all just electricity and binary bits, nothing new here...

/s?

Re: The Unreliability of LLMs and What Lies Ahead

#26

Post by bluefirebrand » Mon, Jun 02, 2025, 7:39 PM UTC

keybrd-intrrpt wrote: post #20

bluefirebrand wrote: post #15

Earlier quoted context omitted.

Sure, but in both cases you are running a real risk of producing incorrect data If you're a product lead and you ask an LLM to produce a script that gets that output, you still should verify the output is correct Otherwise you run a real risk of seeming like an idiot later when you give a report on "tickets closed in the past week" and your data is completely wrong. "Why hasn't John closed any tickets this week? Is h…

> you still should verify the output is correct And that's a problem with the workflow, not a problem with the LLM. It's no different than verifying the information from your Google search or the Stack Overflow answer you found works. But for some reason there are people that have higher expectations of LLM output.

People aren't trying to produce entire codebases in 10 minutes using Stack Overflow, or giving it free reign to refactor the entire codebase

Re: The Unreliability of LLMs and What Lies Ahead

#27

Post by bluefirebrand » Mon, Jun 02, 2025, 7:42 PM UTC

AlienRobot wrote: post #23

I'm no AI fan, but articles talking about the shortcomings of LLM's seem to have to be complaining that forks aren't good for drinking soup. Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry. For the love of God. It's not actual intelligence. This isn't hard. It just randomly spits out text. Use it for what it's good at instead. Text. Instead of hunting for how to do things in program…

> Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry

But use them to do more important things that require more precision and accuracy?

No thanks

Re: The Unreliability of LLMs and What Lies Ahead

#28

Post by mjr00 » Mon, Jun 02, 2025, 7:44 PM UTC

bluefirebrand wrote: post #15

mjr00 wrote: post #12

Earlier quoted context omitted.

It's also what you're writing. The GP's commenter's bio shows they're a product lead, not a full-time software developer. To make some broad assumptions about what kind of code they're talking about: using an LLM for "write me a Python script that queries the Jira API for all tickets closed in the past week" is a much different task from "change the code in our 15 year old in-house accounting software to handle these…

Sure, but in both cases you are running a real risk of producing incorrect data If you're a product lead and you ask an LLM to produce a script that gets that output, you still should verify the output is correct Otherwise you run a real risk of seeming like an idiot later when you give a report on "tickets closed in the past week" and your data is completely wrong. "Why hasn't John closed any tickets this week? Is h…

You're absolutely right. You need to verify the script works, and you need to be able to read the code to see what it's actually doing and if it passes the smell test (as a sibling commenter said, the same way you would for a code snippet off StackOverflow). But ultimately for these bits which are largely rote "take data from API, transform into data format X" tasks, LLMs do a great job getting at least 95% of the way there, in my experience. In a lot of ways they're the perfect job for LLMs: most of the work is just typing (as in, pressing buttons on a keyboard) and passing the right arguments to an API, so why not outsource that to an LLM and verify the output?

The challenge comes when dealing with larger systems. Like an LLM might suggest Library A for accomplishing a task, but if your codebase already has Library B for that already, or maybe Library A but a version from 2020 with a different API, you need to make judgment calls about the right approach to take, and the LLM can't help you there. Same with code style, architecture, how future-proof-but-possibly-YAGNI you want your design to be, etc.

I don't think "vibe coding" or making large changes across big code bases really works (or will ever really work), but I do think LLMs are useful for isolated tasks and it's a mistake to totally dismiss them.

Re: The Unreliability of LLMs and What Lies Ahead

#29

Post by batshit_beaver » Mon, Jun 02, 2025, 7:45 PM UTC

bluefirebrand wrote: post #27

AlienRobot wrote: post #23

I'm no AI fan, but articles talking about the shortcomings of LLM's seem to have to be complaining that forks aren't good for drinking soup. Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry. For the love of God. It's not actual intelligence. This isn't hard. It just randomly spits out text. Use it for what it's good at instead. Text. Instead of hunting for how to do things in program…

> Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry But use them to do more important things that require more precision and accuracy? No thanks

You use LLMs to _discover_ how to approach important problems. You don't necessarily need to use the output verbatim. Same as StackOverflow and Google.

Re: The Unreliability of LLMs and What Lies Ahead

#30

Post by CuriouslyC » Mon, Jun 02, 2025, 7:47 PM UTC

foobiekr wrote: post #18

jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

Charitably, your low expectations are probably the source of your finding them acceptable. It’s also possible - and you should not take this as an insult, it’s just the way it is - you may not know enough about the subjects of your interactions to really spot how wrong they are. However the cases you list - brainstorming - don’t really care about wrong answers. Coding is in the eye of the beholder, but for anything t…

The code that the best frontier models produce is definitely good if you prompt it with what you believe "good" means, with the caveat that code quality depends heavily on the language -- Python, Typescript/Javascript, Java and C are quite good, Rust, C++ and Go tend to be decent to weak depending on the specific model, and other languages are poor.