Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

61–70 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#61

Post by bluefirebrand »

koakuma-chan wrote: post #32
thorum wrote: post #13

Good article. Agree that general unreliability will continue to be an issue since it's fundamental to how LLMs work. However, it would surprise me if there was still a significant gap between single-turn and multi-turn performance in 18 months. Judging by improvements in the last few frontier model releases, I think the top AI labs have finally figured out how to train for multi-turn and agentic capabilities (likely…

They’re reliable enough for many use cases

What this should be doing is exposing how those use cases are faulty, if they can accept such inconsistent and poorly defined outputs

Re: The Unreliability of LLMs and What Lies Ahead

#62

Post by rienbdj »

wintermutestwin wrote: post #50
josefritzishere wrote: post #2

It's hard to say "never" in technology. History isn't really on your side. However, LLMs have largely proven to be good at things computers were are already good at: repetitive tasks, parallel processing, and data analysis. There's nothing magical about an LLM that seems to be defeating the traditional paradigm. Increasingly I lean toward an implosion of the hype cycle for AI.

What I don’t understand is, how can a liar be good at data analysis?

If you give an LLM the data in the prompt and then ask it to extract information from that data it does pretty well. This is the premise of RAG. Where LLMs do poorly is when you ask it for information you haven’t given it.

Re: The Unreliability of LLMs and What Lies Ahead

#63

Post by fellowniusmonk »

whyowhy3484939 wrote: post #43
fellowniusmonk wrote: post #16

Earlier quoted context omitted.

I really don't understand people who are down on LLM. In terms of code output. I have gone from the productivity of being a Sr. Engineer to a team with .8 of a Sr. Engineer, 5 Jr. Engineers and One dude solely dedicated to reading/creating documentation. Unlike a lot of my fellow engineers who are also from traditional CS backgrounds and haven't worked in revenue restricted startup environments, I also have been VERY…

There is this interesting thing called the Paradox of Automation where increasing automation increases the importance of human intervention. We are trying this out on a societal level. It will be.. interesting, to say the least. Also, congratulations on becoming a team. I sure hope you have the mental bandwidth to check all that output carefully. If so, doubly congrats, because you might be the smartest human that ev…

I appreciate you're incredulity and snark! Dismissing without engagement is a fun ability to exercise. I look forward to talking past each other going forward :-)

HackerNews typically doesn't appreciate and will ban accounts for that type of engagement as it is just personal and not a factual wrestling with the point of discussion, I see you are new here and I would encourage you to not continue to engage in the patterns you show.

At core, I think perhaps we have a different interpretation of what 20% of a Sr. Engineer can accomplish and what Jr. Devs are capable of accomplishing.

To be fair to your point, I think one of the enablers is that I actually enjoy working longer hours now so my net time engaging with code has gone up as well.

But I'm from the old school and I've always preferred time in code vs having outside hobbies, that's been true since the 90s.

I find code reviews relaxing and enjoyable and not particularly mentally taxing for 90% of what a decent jr. dev writes. I find it a nice little break from working on problems that can actually be classified as "hard".

Coincidentally, I've worked in human in the loop automation for quite a long time, making Sr. individuals more efficient with their time and removing busy work has been a big focus.

There is a lot in that space to consider from a human factors perspective, the intersection of creation vs editing is a big one, decomposing problems for sure, each individual seems to have different capabilities and natural bents in that regard. I've long been a thought dump and edit person and that's part of what I attribute my high personal productivity to.

Re: The Unreliability of LLMs and What Lies Ahead

#65

Post by mjburgess »

I think I'm settling on a "Gell-mann Amnesia" explanation of why people are so rabidly committed to the "acceptable veracity" of LLM output. When you don't know the facts, you're easily mislead by plausible-sounding analysis, and having been mislead -- a certain default prejudice to existing beliefs takes over. There's a significant asymmetry of effort in belief change vs. acquisition. I think there's also an ego-protection effect here too: if I have to change my belief then I was wrong.

There a socratically-minded people who are more addicted to that moment of belief change, and hence overall vastly more sceptical -- but I think this attitude is extremely marginal. And probably requires a lot of self-training to be properly inculcated into it.

In any case, with LLMs, people really seem to hate the idea that their beliefs about AI and their reliance of LLM output could be systematically mistaken. All the while, when shown output in an area of their expertise, realising immediately that its full of mistakes.

This, of course, makes LLMs a uniquely dangerous force in the health of our social knowledge-conductive processes.

Re: The Unreliability of LLMs and What Lies Ahead

#66

Post by bluefirebrand »

baxtr wrote: post #57
worik wrote: post #49

LLMs are a tool to extend human capabilities. They are not intelligent agents that can replace humans Not very hard to understand, except it seems to be

This. 100%. I think and say this all the time. But people keep continue to say that AI will take all our jobs and I’m so utterly confused by this. Sometimes I wonder if I have gone mad or everyone else.

Companies are salivating over the idea of cutting staff and replacing them with AI tools, so it's not exactly farfetched to think AI might lead to a lot of unemployment, at least for a while

Every type of automation ever invented has led to massive job cuts and yes, some sectors actually did not ever recover

Re: The Unreliability of LLMs and What Lies Ahead

#67

Post by foobiekr »

strangattractor wrote: post #8
jmathai wrote: post #3

My experience with LLm-based chat is so different from what the article (and some friends) describe. I use LLM chat for a wide range of tasks including coding, writing, brainstorming, learning, etc. It’s mostly right enough. And so my usage of it has only increased and expanded. I don’t know how less right it needs to be or how often to reduce my usage. Honestly, I think it’s hard to change habits and LLM chat, at it…

IMHO it's a great summarizing search engine. I now don't have to click on a link to go to that original source - Gemini just hands me a useful summary. Ask AI to do something specific that requires GI (General Intelligence) your milage may vary. So as OpenAI and Google suck in all your content (creators) you are going to find yourself derive less and less revenue generated by visits to your site. Just sayin.

Gemini routinely inaccurately reports the contents in the summary. I have found it actually reversing things on a regular basis. The summary says no and the source says yes.

Re: The Unreliability of LLMs and What Lies Ahead

#68

Post by foobiekr »

CuriouslyC wrote: post #30
foobiekr wrote: post #18

Earlier quoted context omitted.

Charitably, your low expectations are probably the source of your finding them acceptable. It’s also possible - and you should not take this as an insult, it’s just the way it is - you may not know enough about the subjects of your interactions to really spot how wrong they are. However the cases you list - brainstorming - don’t really care about wrong answers. Coding is in the eye of the beholder, but for anything t…

The code that the best frontier models produce is definitely good if you prompt it with what you believe "good" means, with the caveat that code quality depends heavily on the language -- Python, Typescript/Javascript, Java and C are quite good, Rust, C++ and Go tend to be decent to weak depending on the specific model, and other languages are poor.

The C output is absolutely terrible. I cannot fathom an experienced C coder who has found otherwise for anything non trivial. The code is full of things like return from stack, poor buffer size discipline, etc.

Re: The Unreliability of LLMs and What Lies Ahead

#69

Post by empath75 »

bluefirebrand wrote: post #45
yongjik wrote: post #36

Earlier quoted context omitted.

When you employ your developers at $200K/yr you won't trust them to tell you the first one hundred digits of pi, but you'll trust them with your business logic, which is much more important and mission-critical to you. Same thing.

The difference is that (hopefully) your employee is honest enough to say "I do not know the first 100 digits of Pi offhand but I can find out" An LLM will happily produce a string of 100 digits that might be the first 100 digits of Pi, might be some known sequence of 100 digits in Pi but not the first 100, or might be 100 random digits that have nothing to do with Pi

I was actually curious about this and chatgpt actually accurately and very slowly gave me the first 100 digits of pi one digit at a time. I have _no idea_ how that worked, it did not search, nor did it run code. As far as I can tell, it pulled it straight out of it's own model.

If I ask it to use python, it writes and executes the code _much_ more quickly, same if I ask it to search.

Re: The Unreliability of LLMs and What Lies Ahead

#70

Post by DanHulton »

AlienRobot wrote: post #23

I'm no AI fan, but articles talking about the shortcomings of LLM's seem to have to be complaining that forks aren't good for drinking soup. Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry. For the love of God. It's not actual intelligence. This isn't hard. It just randomly spits out text. Use it for what it's good at instead. Text. Instead of hunting for how to do things in program…

The thing is, I have spent the last year being told that I will VERY SOON be able to use a fork to drink soup, and better than any spoon has ever been able to, and in fact pretty soon spoons will be completely outclassed anyway, and I'M the idiot for doubting this.

Articles like this are still very much needed, to push back against that narrative, regularly, until it DOES become as obvious to everyone as it is to you.

Post reply on HN