Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I genuinely don't understand why some people are still bullish about LLMs

twitter.com

Post reply on HN
Print view View original

361–370 of 1001 posts

Re: I genuinely don't understand why some people are still bullish about LLMs

#361

Post by uoaei »

inopinatus wrote: post #290
Animats wrote: post #178

Earlier quoted context omitted.

> inbuilt into the models of all LLMs is that they'll fabricate a lot. Still the elephant in the room. We need an AI technology that can output "don't know" when appropriate. How's that coming along?

Unfortunately they are trained first and foremost as plausibility engines. The central dogma is that plausibility will (with continuing progress & scale) converge towards correctness, or "faithfulness" as it's sometimes called in the literature. This remains very far from proven. The null hypothesis that would be necessary to reject, therefore, is a most unfortunate one, viz. that by training for plausibility we are…

One man's "plausibility" is another person's "barely reasoned bullshit". I think you're being generous, because LLMs explicitly don't deal in facts, they deal in making stuff up that is vaguely reminiscent of fact. Only a few companies are even trying to make reasoning (as in axioms-cum-deductions, i.e., logic per se) a core part of the models, and they're really struggling to hand-engineer the topology and methodology necessary for that to work roughly as facsimile of technical reasoning.

Re: I genuinely don't understand why some people are still bullish about LLMs

#362

Post by [deleted] »

gilbetron wrote: post #281
Terr_ wrote: post #267

Earlier quoted context omitted.

It's like computer graphics and VR: Amazing advances over the years, very impressive, fun, cool, and by no means a temporary fad... ... But I do not believe we're on the cusp of a Lawnmower-Man future where someone's Metaverse eats all the retail-conference-halls and movie-theaters and retail-stores across the entire globe in an unbridled orgy of mind-shattering investor returns . Similarly, LLMs are neat and have so…

Ah yes, the "computer graphics, which has generated billions upon billions of revenue and change are just a fad" argument. What next, "This Internet thing was just a fad" or "The industrial age was a fad"?

[deleted]

Re: I genuinely don't understand why some people are still bullish about LLMs

#363

Post by mac-mc »

gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

LLMs are better at formally verifiable tasks like coding, also coding makes more money on a pure demand basis so development for it gets more resources. In descriptive science fields, it's not great because science fields don't generate a lot of text compared to other things, so the training data is dwarfed by the huge corpus of general internet text. The software industry created the internet and loves using it, so they have published a lot more text in comparison. It can be really bad in bio for example.

Re: I genuinely don't understand why some people are still bullish about LLMs

#364

Post by sinuhe69 »

Funny how most of the counter comments here used the form "my experience is different/it's amazing!" and then listed activities that are completely different from what Sabine listed :)

We really should stop reinforcing our echo bubbles and learn from other people. And sometimes be cool in the face of criticism.

Re: I genuinely don't understand why some people are still bullish about LLMs

#365

Post by trial3 »

nonethewiser wrote: post #355
cogman10 wrote: post #161

Earlier quoted context omitted.

> Wah, it can't write code like a Senior engineer with 20 years of experience! No, that's not my problem with it. My problem with it is that inbuilt into the models of all LLMs is that they'll fabricate a lot. What's worse, people are treating them as authoritative. Sure, sometimes it produces useful code. And often, it'll simply call the "doTheHardPart()" method. I've even caught it literally writing the wrong algor…

> What's worse, people are treating them as authoritative. … I've both seen online and heard people quote LLM output as if it were authoritative. Thats not an LLM problem. But indeed quite bothersome. Dont tell me what Chatgpt told you. Tell me what you know. Maybe you got it from ChatGPT and verified it. Great. But my jaw kind of drops when people cite an LLM and just assume it’s correct.

It might not be an LLM problem, but it’s an AI-as-product problem. I feel like every major player’s gamble is that they can cement distinct branding and model capabilities (as perceived by the public) faster than the gradual calcification of public AI perception catches up with model improvements - every time a consumer gets burned by AI output in even small ways, the “AI version of Siri/Alexa only being used for music and timers” problem looms a tiny, tiny bit larger.

Re: I genuinely don't understand why some people are still bullish about LLMs

#366

Post by worthless-trash »

cogman10 wrote: post #161
gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

> Wah, it can't write code like a Senior engineer with 20 years of experience! No, that's not my problem with it. My problem with it is that inbuilt into the models of all LLMs is that they'll fabricate a lot. What's worse, people are treating them as authoritative. Sure, sometimes it produces useful code. And often, it'll simply call the "doTheHardPart()" method. I've even caught it literally writing the wrong algor…

I saw someone saying 80% of doctors believe that LLM's are trustworthy consultation partners.

Code created by LLM's doesnt compile, hallucinated API's.. invalid syntax and completely broken logic, why would you trust it with someones life !

Re: I genuinely don't understand why some people are still bullish about LLMs

#367

Post by eviks »

gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

Of course you'd be confused if your transform a list of basic fails people complain about into

> And people are like, "Wah, it can't write code like a Senior engineer with 20 years of experience!"

But LLMs should be good enough to resolve this confusion, ask them!

Re: I genuinely don't understand why some people are still bullish about LLMs

#368

Post by anxoo »

Animats wrote: post #178
cogman10 wrote: post #161

Earlier quoted context omitted.

> Wah, it can't write code like a Senior engineer with 20 years of experience! No, that's not my problem with it. My problem with it is that inbuilt into the models of all LLMs is that they'll fabricate a lot. What's worse, people are treating them as authoritative. Sure, sometimes it produces useful code. And often, it'll simply call the "doTheHardPart()" method. I've even caught it literally writing the wrong algor…

> inbuilt into the models of all LLMs is that they'll fabricate a lot. Still the elephant in the room. We need an AI technology that can output "don't know" when appropriate. How's that coming along?

would you rather the LLM make up something that sounds right when it doesn't know, or would you like it to claim "i don't know" for tasks it actually can figure out? because presumably both happen at some rate, and if it hallucinates an answer i can at least check what that answer is or accept it with a grain of salt.

nobody freaks out when humans make mistakes, but we assume our nascent AIs, being machines, should always function correctly all the time

Re: I genuinely don't understand why some people are still bullish about LLMs

#369

Post by jjani »

mystified5016 wrote: post #86
turtletontine wrote: post #30

Earlier quoted context omitted.

I literally had a developer of an open source package I’m working with tell me “yeah that’s a known problem, I gave up on trying to fix it. You should just ask ChatGPT to fix it, I bet it will immediately know the answer.” Annoying response of course. But I’d never used an LLM to debug before, so I figured I’d give it a try. First: it regurgitated a bunch of documentation and basic debugging tips, which might have ac…

The biggest thing I've found is that if you give any hint at all as to what you think the problem is, the LLM will immediately and enthusiastically agree, no matter how wildly incorrect your suggestion is. If I give it all my information and add "I think the problem might be X, but I'm not sure", the LLM always agrees that the problem is X and will reinterpret everything else I've said to 'prove' me right. Then the c…

> The biggest thing I've found is that if you give any hint at all as to what you think the problem is, the LLM will immediately and enthusiastically agree, no matter how wildly incorrect your suggestion is.

I'm seeing this sentiment a lot in these comments, and frankly it shows that very few here have actually gone and tried the variety of models available. Which is totally fine, I'm sure they have better stuff to do, you don't have to keep up with this week's hottest release.

To be concrete - the symptom you're talking about is very typical of Claude (or earlier GPT models). o3-mini is much less likely to do this.

Secondly, prompting absolutely goes a huge way to avoiding that issue. Like you're saying - if you're not sure, don't give hints, keep it open-minded. Or validate the hint before starting, in a separate conversation.

Re: I genuinely don't understand why some people are still bullish about LLMs

#370

Post by [deleted] »

inopinatus wrote: post #290
Animats wrote: post #178

Earlier quoted context omitted.

> inbuilt into the models of all LLMs is that they'll fabricate a lot. Still the elephant in the room. We need an AI technology that can output "don't know" when appropriate. How's that coming along?

Unfortunately they are trained first and foremost as plausibility engines. The central dogma is that plausibility will (with continuing progress & scale) converge towards correctness, or "faithfulness" as it's sometimes called in the literature. This remains very far from proven. The null hypothesis that would be necessary to reject, therefore, is a most unfortunate one, viz. that by training for plausibility we are…

[deleted]
Post reply on HN