Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

91–100 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#91

Post by elictronic »

ar813 wrote: post #59

If I take a step back and think back to say a few (or 5) years ago, what LLMs can do is amazing. One has to acknowledge that (or at least, I do). But as a scientist it's been rather interesting to probe the jagged edge and unreliability, including using deep research tools, on any topic I know well. If I read through the reports and summaries it generates, it seems at first glance correct - the jargon is used correct…

I think those in the computer science field see passable results of LLM use with respect to software and papers and start assuming other engineering fields should be easy.

They fail to understand other engineering fields documentation and process are awful. Not that computer science is good because they are even less rigorous.

The difference is other fields don’t log every single change they make into source control and have millions of open source projects to pull from. There aren’t billions of books on engineering to pull from like with language. The information is siloed and those with the keys now know what it’s worth.

Re: The Unreliability of LLMs and What Lies Ahead

#92

Post by uludag »

AlienRobot wrote: post #23

I'm no AI fan, but articles talking about the shortcomings of LLM's seem to have to be complaining that forks aren't good for drinking soup. Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry. For the love of God. It's not actual intelligence. This isn't hard. It just randomly spits out text. Use it for what it's good at instead. Text. Instead of hunting for how to do things in program…

So many times I've asked questions just like this and gotten complete nonsense incorrect answers. In fact, you have no guarantees whatsoever that even the typescript question you asked will always return a sensible answer.

I'm by no means saying that LLMs aren't useful. They're just not reliably useful.

Re: The Unreliability of LLMs and What Lies Ahead

#93

Post by ar813 »

crooked-v wrote: post #86
ar813 wrote: post #59

If I take a step back and think back to say a few (or 5) years ago, what LLMs can do is amazing. One has to acknowledge that (or at least, I do). But as a scientist it's been rather interesting to probe the jagged edge and unreliability, including using deep research tools, on any topic I know well. If I read through the reports and summaries it generates, it seems at first glance correct - the jargon is used correct…

I'm reminded of the whole "vegetative electron microscopy" mess ( https://www.sciencealert.com/a-strange-phrase-keeps-turning-... ).

That's wild! Now I want to go hunting for more such examples..

Re: The Unreliability of LLMs and What Lies Ahead

#94

Post by izabera »

akomtu wrote: post #42

LLMs can't evaluate their own output. LLMs suggest possibilities, but can't evaluate them. Imagine an insane man who is rumbling something smart, but doesn't self-reflect. The evaluation is done against some framework of values that are considered true: the rules of a board game, the language syntax or something else. LLMs also can't fabricate evaluation because the latter is a rather rigid and precise model, a unlik…

oh boy do i have the paper for you https://proceedings.neurips.cc/paper_files/paper/2014/file/f...

Re: The Unreliability of LLMs and What Lies Ahead

#95

Post by yoyohello13 »

fellowniusmonk wrote: post #16
jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

I really don't understand people who are down on LLM. In terms of code output. I have gone from the productivity of being a Sr. Engineer to a team with .8 of a Sr. Engineer, 5 Jr. Engineers and One dude solely dedicated to reading/creating documentation. Unlike a lot of my fellow engineers who are also from traditional CS backgrounds and haven't worked in revenue restricted startup environments, I also have been VERY…

> Those who are on the MOST extreme end of the mathematic side and are linguistically bereft HATE LLM's and effectively cannot use them.

This is an interesting observation. It at least aligns with my experience. I wouldn't say I'm "linguistically bereft" lol, but I do lean more toward the "functional programming is beautiful" side. I even have a degree in math. I'm not totally down on LLM coding, but I do fall more on unfavorable feelings side. I mostly just hate the idea of having a bunch of code I don't fully understand, but also am responsible for.

I do use them, and find them helpful. But the idea of fully giving control of my codebase to LLM agents, like some people are suggesting, repels me.

Re: The Unreliability of LLMs and What Lies Ahead

#96

Post by bluefirebrand »

empath75 wrote: post #76
bluefirebrand wrote: post #7

Earlier quoted context omitted.

> It’s mostly right enough. Honestly this is why your experience is different: your expectations are different (and likely lower). I never find they are "mostly right enough", I find they are "mostly wrong in ways that range from subtle mistakes to extremely incorrect". The more subtly they are wrong, the worse I rate their output actually, because that is what costs me more time when I try to use them I want tools t…

They save me a tremendous amount of time, you just need to be smart about what you try to get them to do. _Busy work_ is what you want to focus on, not anything that takes a ton of domain knowledge and intelligence. Just as an example from today, i had a huge pile of yaml documents that needed to have some transformations done to them -- they were pretty simple and obvious, but I just went into cursor, give it a befo…

> _Busy work_ is what you want to focus on, not anything that takes a ton of domain knowledge and intelligence

Eh..

Maybe that's more of a sign that we shouldn't be doing busywork in the first place

Re: The Unreliability of LLMs and What Lies Ahead

#98

Post by andrewmutz »

brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

What we are seeing with our customers is that LLM errors are a very manageable problem. End users adapt pretty quickly to the idea that AI systems aren't perfect. In many cases AI products are doing tasks that used to be done by humans and these humans were making mistakes too, so the end user is used to the idea that the task will get accomplished with some non-zero error rate.

You just need to build your products in a manner where the user has the ability to easily double check the results whenever they like. Then they can audit as they see fit, in order to get used to the accuracy level and to apply additional scrutiny to cases that are very important to their business.

Re: The Unreliability of LLMs and What Lies Ahead

#99

Post by mdp2021 »

izabera wrote: post #94
akomtu wrote: post #42

LLMs can't evaluate their own output. LLMs suggest possibilities, but can't evaluate them. Imagine an insane man who is rumbling something smart, but doesn't self-reflect. The evaluation is done against some framework of values that are considered true: the rules of a board game, the language syntax or something else. LLMs also can't fabricate evaluation because the latter is a rather rigid and precise model, a unlik…

oh boy do i have the paper for you https://proceedings.neurips.cc/paper_files/paper/2014/file/f...

Sorry, what do GANs have to do with this? It is not the same kind of "evaluation".

And anyway, there is no need to have two networks to iteratively refine output: one suffices (like we naturally are meant to do).

Re: The Unreliability of LLMs and What Lies Ahead

#100

Post by bluefirebrand »

mjr00 wrote: post #53
bluefirebrand wrote: post #37

Earlier quoted context omitted.

> so why not outsource that to an LLM and verify the output? I mean sure, why not. My argument isn't that it doesn't work, it's that it doesn't really save time If you try to have it do big changes you will be swamped reviewing those changes for correctness for a long time while you build a mental model of the work If you have it do small changes, the actual performance improvement is marginal at best, because small…

I disagree that it doesn't save time for some classes of problems. As a concrete recent example, I had to write a Python script which checked for any postgres tables where the primary key was of type 'INT' and print out the max value of the ID for each table. I know broadly how to do this, but I'd have to double check which information_schema table to use, the right names of the columns to use, etc. Plus a refresher…

I don't see a lot of value in "saving 10-20 minutes here and there" tbh

Especially since I'm not ever likely to see any benefit from my employer for that extra productivity

Post reply on HN