Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

11–20 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#11

Post by light_hue_1 »

jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

> It’s mostly right enough

What do you use it for?

In my space, "mostly right enough" isn't useful. Particularly when that means that the errors are subtle and I might miss them. I can't write whitepapers that tell people to do things that would result in major losses.

Re: The Unreliability of LLMs and What Lies Ahead

#12

Post by mjr00 »

bluefirebrand wrote: post #7
jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

> It’s mostly right enough. Honestly this is why your experience is different: your expectations are different (and likely lower). I never find they are "mostly right enough", I find they are "mostly wrong in ways that range from subtle mistakes to extremely incorrect". The more subtly they are wrong, the worse I rate their output actually, because that is what costs me more time when I try to use them I want tools t…

It's also what you're writing. The GP's commenter's bio shows they're a product lead, not a full-time software developer. To make some broad assumptions about what kind of code they're talking about: using an LLM for "write me a Python script that queries the Jira API for all tickets closed in the past week" is a much different task from "change the code in our 15 year old in-house accounting software to handle these tariffs", both in terms of the code that gets written as well as the consequences of the LLM getting it wrong.

To be clear this isn't a knock on anyone's work, but it does seem to be a source of why "pro-LLM" and "anti-LLM" groups tend to talk past each other.

Re: The Unreliability of LLMs and What Lies Ahead

#13

Post by thorum »

Good article. Agree that general unreliability will continue to be an issue since it's fundamental to how LLMs work. However, it would surprise me if there was still a significant gap between single-turn and multi-turn performance in 18 months. Judging by improvements in the last few frontier model releases, I think the top AI labs have finally figured out how to train for multi-turn and agentic capabilities (likely RL) and just need to scale this up.

Re: The Unreliability of LLMs and What Lies Ahead

#14

Post by kenjackson »

nomel wrote: post #9
jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

From what I can tell, rather than a simple difference in expectation (which could explain your positive experience vs others), it seems to be a "comfort within uncertainty" difference that, from what I can tell, is a personality trait! You're comfortable with the uncertainty, and accommodate it in your use and expectations. You're left feeling good about the experience, within that uncertainty. Others are repelled by…

I wonder if this is like dishwasher usage. As a kid growing up we never used the dishwasher. It was just the drying rack. The reason was you had to rinse off the big stuff anyways, and then the resulting quality of dishwashing was poor in it. You'd often get a fork with rice stuck between it still, which was unacceptable.

As a grown up now I use a dishwasher for everything that is permitted to go in it. I still have to rinse off plates first, and occasionally I do see rice between a fork that I have to then clean manually. But I'm not comfortable knowing that it won't clean as well as I could by hand, but it does a good enough job -- and in some ways a much better job (it uses much hotter water than I do by hand). I don't know if my mom could ever really be comfortable with it though.

Re: The Unreliability of LLMs and What Lies Ahead

#15

Post by bluefirebrand »

mjr00 wrote: post #12
bluefirebrand wrote: post #7

Earlier quoted context omitted.

> It’s mostly right enough. Honestly this is why your experience is different: your expectations are different (and likely lower). I never find they are "mostly right enough", I find they are "mostly wrong in ways that range from subtle mistakes to extremely incorrect". The more subtly they are wrong, the worse I rate their output actually, because that is what costs me more time when I try to use them I want tools t…

It's also what you're writing. The GP's commenter's bio shows they're a product lead, not a full-time software developer. To make some broad assumptions about what kind of code they're talking about: using an LLM for "write me a Python script that queries the Jira API for all tickets closed in the past week" is a much different task from "change the code in our 15 year old in-house accounting software to handle these…

Sure, but in both cases you are running a real risk of producing incorrect data

If you're a product lead and you ask an LLM to produce a script that gets that output, you still should verify the output is correct

Otherwise you run a real risk of seeming like an idiot later when you give a report on "tickets closed in the past week" and your data is completely wrong. "Why hasn't John closed any tickets this week? Is he slacking off?"... "What he closed more tickets than anyone..." And then it turns out that the unreliable LLM script excluded him for whatever reason

Of course I understand that people are not going to actually be this careful, because more and more people are trusting LLM output without verifying it. Because it's "right enough" that we are becoming complacent

Re: The Unreliability of LLMs and What Lies Ahead

#16

Post by fellowniusmonk »

jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

I really don't understand people who are down on LLM.

In terms of code output. I have gone from the productivity of being a Sr. Engineer to a team with .8 of a Sr. Engineer, 5 Jr. Engineers and One dude solely dedicated to reading/creating documentation.

Unlike a lot of my fellow engineers who are also from traditional CS backgrounds and haven't worked in revenue restricted startup environments, I also have been VERY into interpreted languages like ruby in the past.

Now compiled languages are even better, I think from a velocity perspective compiled languages are now incredibly on par for prototyping velocity and have had their last weakness removed.

It's both exciting and scary, I can't believe how people are still sleep walking in this environment and don't realize we are in a different world. Once again the human inability to "gut reason" about exponentials is going to screw us all over.

One terribly overlooked thing I've noticed that I think explains the differing takes. Foundation of my position here: https://www.nature.com/articles/s41598-020-60661-8

Within the population that writes code there are a small number of successful people who approach the topic in a ~purely mathematical approach, and a small number of successful people that approach writing code in a ~purely linguistic approach. Most people fall somewhere in the middle.

Those who are on the MOST extreme end of the mathematic side and are linguistically bereft HATE LLM's and effectively cannot use them.

My guess is that HN population will tend to show stronger reactions against LLM's because it was heavily seeded with functional programmers which I think has a concentration of the successful extremely math focused. I worked for several years in a purely functional shop and that was my observation: Elixir, Haskell, Ramda.

Just my speculation.

Re: The Unreliability of LLMs and What Lies Ahead

#17

Post by hooverd »

strangattractor wrote: post #8
jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

IMHO it's a great summarizing search engine. I now don't have to click on a link to go to that original source - Gemini just hands me a useful summary. Ask AI to do something specific that requires GI (General Intelligence) your milage may vary. So as OpenAI and Google suck in all your content (creators) you are going to find yourself derive less and less revenue generated by visits to your site. Just sayin.

DuckDuckGo, which uses Bing I think, now has Bing's AI summaries instead of the goddamn content in search results, which makes evaluating the search results at a glance useless!

Re: The Unreliability of LLMs and What Lies Ahead

#18

Post by foobiekr »

jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

Charitably, your low expectations are probably the source of your finding them acceptable.

It’s also possible - and you should not take this as an insult, it’s just the way it is - you may not know enough about the subjects of your interactions to really spot how wrong they are.

However the cases you list - brainstorming - don’t really care about wrong answers.

Coding is in the eye of the beholder, but for anything that isn’t junk glue code, scripts or low-complexity web stuff, I find the output of LLMs just short of horrendous.

Re: The Unreliability of LLMs and What Lies Ahead

#19

Post by bluefirebrand »

kenjackson wrote: post #14
nomel wrote: post #9

Earlier quoted context omitted.

From what I can tell, rather than a simple difference in expectation (which could explain your positive experience vs others), it seems to be a "comfort within uncertainty" difference that, from what I can tell, is a personality trait! You're comfortable with the uncertainty, and accommodate it in your use and expectations. You're left feeling good about the experience, within that uncertainty. Others are repelled by…

I wonder if this is like dishwasher usage. As a kid growing up we never used the dishwasher. It was just the drying rack. The reason was you had to rinse off the big stuff anyways, and then the resulting quality of dishwashing was poor in it. You'd often get a fork with rice stuck between it still, which was unacceptable. As a grown up now I use a dishwasher for everything that is permitted to go in it. I still have…

Imagine if the advice for Dishwasher usage mirrored the advice for AI

"You have to iterate on the output to get good results"

Just keep running that dishwasher until they're clean! If you run it and they're still dirty, load it up with soap and try again!

Re: The Unreliability of LLMs and What Lies Ahead

#20

Post by keybrd-intrrpt »

bluefirebrand wrote: post #15
mjr00 wrote: post #12

Earlier quoted context omitted.

It's also what you're writing. The GP's commenter's bio shows they're a product lead, not a full-time software developer. To make some broad assumptions about what kind of code they're talking about: using an LLM for "write me a Python script that queries the Jira API for all tickets closed in the past week" is a much different task from "change the code in our 15 year old in-house accounting software to handle these…

Sure, but in both cases you are running a real risk of producing incorrect data If you're a product lead and you ask an LLM to produce a script that gets that output, you still should verify the output is correct Otherwise you run a real risk of seeming like an idiot later when you give a report on "tickets closed in the past week" and your data is completely wrong. "Why hasn't John closed any tickets this week? Is h…

> you still should verify the output is correct

And that's a problem with the workflow, not a problem with the LLM.

It's no different than verifying the information from your Google search or the Stack Overflow answer you found works. But for some reason there are people that have higher expectations of LLM output.

Post reply on HN