Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

51–60 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#51

Post by candiddevmike »

akomtu wrote: post #42

LLMs can't evaluate their own output. LLMs suggest possibilities, but can't evaluate them. Imagine an insane man who is rumbling something smart, but doesn't self-reflect. The evaluation is done against some framework of values that are considered true: the rules of a board game, the language syntax or something else. LLMs also can't fabricate evaluation because the latter is a rather rigid and precise model, a unlik…

Isn't this kind of the hope/dream of multi-agent systems where one LLM "coordinates" among others or checks the responses? In my experience it works about as well as you're describing.

Re: The Unreliability of LLMs and What Lies Ahead

#52

Post by ToucanLoucan »

josefritzishere wrote: post #2

It's hard to say "never" in technology. History isn't really on your side. However, LLMs have largely proven to be good at things computers were are already good at: repetitive tasks, parallel processing, and data analysis. There's nothing magical about an LLM that seems to be defeating the traditional paradigm. Increasingly I lean toward an implosion of the hype cycle for AI.

LLMs are a legitimate technology with legitimate applications. However in a desperate bid for a new iPhone moment to assure Wall Street that the fantasy of infinite growth in a finite world is possible, they have utterly lost the plot regarding what statistical analysis of words at scale is capable of doing. Useless? Far from it. The basis for a 300 billion company with no meaningful products after almost a decade working on it? I have doubts.

I can't fathom a future where OpenAI for sure doesn't eat dirt, with Anthropic likely not far behind it. nVidia will likely come out fine, since it still has gamers to disappoint, and the infrastructure build out that did occur will crater the cost of GPUs at scale for smaller, smarter companies to take advantage of. So it will likely still kick around, but as another technology, not the second coming of Cyber Christ as it's been hyped to be.

Re: The Unreliability of LLMs and What Lies Ahead

#53

Post by mjr00 »

bluefirebrand wrote: post #37
mjr00 wrote: post #28

Earlier quoted context omitted.

You're absolutely right. You need to verify the script works, and you need to be able to read the code to see what it's actually doing and if it passes the smell test (as a sibling commenter said, the same way you would for a code snippet off StackOverflow). But ultimately for these bits which are largely rote "take data from API, transform into data format X" tasks, LLMs do a great job getting at least 95% of the wa…

> so why not outsource that to an LLM and verify the output? I mean sure, why not. My argument isn't that it doesn't work, it's that it doesn't really save time If you try to have it do big changes you will be swamped reviewing those changes for correctness for a long time while you build a mental model of the work If you have it do small changes, the actual performance improvement is marginal at best, because small…

I disagree that it doesn't save time for some classes of problems.

As a concrete recent example, I had to write a Python script which checked for any postgres tables where the primary key was of type 'INT' and print out the max value of the ID for each table. I know broadly how to do this, but I'd have to double check which information_schema table to use, the right names of the columns to use, etc. Plus a refresher on direct use of psycopg2 and the cursor API. Plus the typing itself. I just put that query into an LLM and it gave me exactly what I needed, took about 30-60 seconds total. Between the research and typing that's easily 10 minutes saved, maybe closer to 20 really.

And I mean, no, this example isn't worth the $10 trillion or whatever the economy thinks AI is worth, but given that it exists, I'm happy to take advantage of it.

Re: The Unreliability of LLMs and What Lies Ahead

#54

Post by bluefirebrand »

throwacct wrote: post #48
bluefirebrand wrote: post #7

Earlier quoted context omitted.

> It’s mostly right enough. Honestly this is why your experience is different: your expectations are different (and likely lower). I never find they are "mostly right enough", I find they are "mostly wrong in ways that range from subtle mistakes to extremely incorrect". The more subtly they are wrong, the worse I rate their output actually, because that is what costs me more time when I try to use them I want tools t…

This. I use LLMs for some tasks, but for more complex issues, I do it myself. I tried to use it for a project by defining each task as clearly as possible, and I spent weeks trying to come up with something useful. Mind you, I achieved 80% of what I wanted after iterating and "telling" the chat that their answers were wrong, and going over the code to double-check if everything was okay. Now I use it for specific, si…

> Mind you, I achieved 80% of what I wanted after iterating and "telling" the chat that their answers were wrong, and going over the code to double-check if everything was okay

I very often read things like this, and I'm surprised how often the person estimates "around 80%" of the work was good. It feels so perfectly tailored to the Pareto Principal

The LLM does the easy 80% (which we usually say takes 20% of the time anyways). Then the human has to go do the harder remaining 20%, only with a much smaller mental model of how the original 80% is fitting together

Re: The Unreliability of LLMs and What Lies Ahead

#55

Post by liveoneggs »

eterm wrote: post #44

There are jobs out there that have always been unreliable. A classic example is the Travel Agent. This was already a job driven to near-extinction just by Google, but LLMs are a nail in the travel agent coffin. The job was always fuzzy. It was always unreliable. A travel agent recommendation was never a stamp of quality or guarentee of satisfaction. But now, I can ask an LLM to compare and contrast two weeks in the S…

I have used it on three big family vacations already and it's definitely a place where "AI" shines in usefulness. It did recommend some out-of-business hotels and things but the broad strokes were good enough to save hours of work.

Re: The Unreliability of LLMs and What Lies Ahead

#56

Post by ToucanLoucan »

wintermutestwin wrote: post #50
josefritzishere wrote: post #2

It's hard to say "never" in technology. History isn't really on your side. However, LLMs have largely proven to be good at things computers were are already good at: repetitive tasks, parallel processing, and data analysis. There's nothing magical about an LLM that seems to be defeating the traditional paradigm. Increasingly I lean toward an implosion of the hype cycle for AI.

What I don’t understand is, how can a liar be good at data analysis?

It works great if all you're looking for is an output, with not a care for what it is. So if you're trying to generate slop children's books to shit onto Amazon, it's awesome. If you want to give your boss a huge bloated report on your daily activities, works great. If you want to phone in an assignment that doesn't add value to your education, LLM will do that. If you want a header image for your LinkedIn post that you don't want to pay for, generate it. Who cares.

This isn't even an indictment, not really. I'm just reading between the lines here regarding when/how it's used. Nobody with intentionality uses these things. Nobody who CARES what they're making uses these things. And again, I want to emphasize, this is not an attack. There are tons of things I do in my work life that I utterly do not give a shit about, and LLMs have been a blessing for it. Not my code, fuck no. But all the ancillary crap, absolutely.

Re: The Unreliability of LLMs and What Lies Ahead

#57

Post by baxtr »

worik wrote: post #49

LLMs are a tool to extend human capabilities. They are not intelligent agents that can replace humans Not very hard to understand, except it seems to be

This. 100%.

I think and say this all the time. But people keep continue to say that AI will take all our jobs and I’m so utterly confused by this.

Sometimes I wonder if I have gone mad or everyone else.

Re: The Unreliability of LLMs and What Lies Ahead

#58

Post by blharr »

keybrd-intrrpt wrote: post #24
bluefirebrand wrote: post #19

Earlier quoted context omitted.

Imagine if the advice for Dishwasher usage mirrored the advice for AI "You have to iterate on the output to get good results" Just keep running that dishwasher until they're clean! If you run it and they're still dirty, load it up with soap and try again!

That's all new technology though. Dishwashers _were_ like that. What's seemed to change are people's expectations of technology that "just works". When in reality, we are in the infant years of AI/ML and LLMs We're so spoiled by the pace of innovation we're upset it requires a bit of hand-holding while they figure things out.

It's still egregious because the main theme is "Learn how to work with AI so you won't be left behind in the future!" The analogy in that case is to waste time pointlessly learning the quirks of old dishwashers while new dishwashers won't have them in the future.

Re: The Unreliability of LLMs and What Lies Ahead

#59

Post by ar813 »

If I take a step back and think back to say a few (or 5) years ago, what LLMs can do is amazing. One has to acknowledge that (or at least, I do). But as a scientist it's been rather interesting to probe the jagged edge and unreliability, including using deep research tools, on any topic I know well.

If I read through the reports and summaries it generates, it seems at first glance correct - the jargon is used correctly, and physical phenomena referred to mostly accurately. But very quickly I realize that, even with the deep research features and citations, it's making a bunch of incorrect inferences that likely arise from certain concepts (words, really) co-occurring in documents but are actually physically not causally linked or otherwise fundamentally connected. In addition to some strange leading sentences and arguments made, this often ends up creating entirely inappropriate topic headings/ sections connecting things that really shouldn't be together.

One small example of course, but this type of error (usually multiple errors) shows up in both Gemini and OpenAI models, and even with some very specific prompts and multiple turns. And keeps happening for topics in the fields I work in in the physical sciences and engineering. I'm not sure one could RL hard enough to correct this sort of thing (and it is not likely worth the time and money), but perhaps my imagination is limited.

Re: The Unreliability of LLMs and What Lies Ahead

#60

Post by yegg »

hooverd wrote: post #17
strangattractor wrote: post #8

Earlier quoted context omitted.

IMHO it's a great summarizing search engine. I now don't have to click on a link to go to that original source - Gemini just hands me a useful summary. Ask AI to do something specific that requires GI (General Intelligence) your milage may vary. So as OpenAI and Google suck in all your content (creators) you are going to find yourself derive less and less revenue generated by visits to your site. Just sayin.

DuckDuckGo, which uses Bing I think, now has Bing's AI summaries instead of the goddamn content in search results, which makes evaluating the search results at a glance useless!

For what it's worth, we produce our own summaries, and you can turn them off if you don't like them. We also offer noai.duckduckgo.com, which turns all of our AI features off automatically.
Post reply on HN