Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

181–190 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#181

Post by biophysboy »

hbs18 wrote: post #116
jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

> The basic design is non-deterministic Is it? I thought an LLM was deterministic provided you run the exact same query on exact same hardware at a temperature of 0.

My understanding is that it selects from a probability distribution. Raising the temperature merely flattens that distribution, Boltzmann factor style

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#182

Post by dlisboa »

Davidzheng wrote: post #10
jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

lol humans are non-deterministic too

Which is why every tool that is better than humans at a certain task are deterministic.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#183

Post by MangoToupe »

y0eswddl wrote: post #115
MangoToupe wrote: post #109

Earlier quoted context omitted.

> Reflect a moment over the fact that LLMs currently are just text generators. You could say the same thing about humans.

No, you actually can't. Humans existed for 10s to 100s of thousands of years without text. or even words for that matter.

I disagree: it is language that makes us human.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#184

Post by touristtam »

nickdothutton wrote: post #44

- Claude, please optimise the project for performance. o Claude goes away for 15 minutes, doesn't profile anything, many code changes. o Announces project now performs much better, saving 70% CPU. - Claude, test the performance. o Performance is 1% _slower_ than previous. - Claude, can I have a refund for the $15 you just wasted? o [Claude waffles], "no".

The last bit, in my limited experience:

> Claude: sorry you have to want until XX:00 as you have run out of credit.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#185

Post by vlowther »

MangoToupe wrote: post #109
motoboi wrote: post #98

Earlier quoted context omitted.

Reflect a moment over the fact that LLMs currently are just text generators. Also that the conversational behavior we see it’s just examples of conversations that we have the model to mimic so when we say “System: you are a helpful assistant. User: let’s talk. Assistant:” it will complete the text in a way that mimics a conversation?. Yeah, we improved over that using reinforcement learning to steer the text generati…

> Reflect a moment over the fact that LLMs currently are just text generators. You could say the same thing about humans.

No, you cannot. Our abstract language abilities (especially the written word part) are a very thin layer on top of hundreds of millions of years of evolution in an information dense environment.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#186

Post by cindyllm »

jacquesm wrote: post #151
coldtea wrote: post #145

Earlier quoted context omitted.

> LLMs are text model, not world models and that is the root cause of the problem. Is it though? In the end, the information in the training texts is a distilled proxy for the world, and the weighted model ends up being a world model, just an once-removed one. Text is not that different to visual information in that regard (and humans base their world model on both). > Not having a world model is a massive disadvanta…

I really think it is, this is the exact same thing that keeps going wrong in these conversations over-and-over again. There simply is no common sense, none at all, just a likelihood of applicability. To the point that I even wonder how it is possible to get such basic stuff for which there is an insane amount of support wrong. I've had an hour long session which essentially revolved around why the landing gear of an…

[dead]

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#187

Post by lkbm »

jacquesm wrote: post #130
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

If you're paying per token then there is a big business incentive for the counterparty to burn tokens as much as possible.

If I'll pay to get a fixed result, sure. I'd expect a Jevons paradox effect: if LLMs got me results twice as fast for the same cost, I'm going to use it more and end up paying more in total.

Maximizing the utility of your product for users is usually the winning strategy.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#188

Post by amelius »

BubbleRings wrote: post #123
wlesieutre wrote: post #77

Earlier quoted context omitted.

My favorite is when it bounces back and forth between the same two wrong answers, each time admitting that the most recent answer is wrong and going back to the previous wrong answer. Doesn't matter if you tell it "that's not correct and neither is ____ so don't try that instead," it likes those two answers and it's going to keep using them.

Ha! Just experienced this. It was very frustrating.

They really need to add a "punish the LLM" button.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#189

Post by Mockapapella »

I wrote about something like this a couple months ago: https://thelisowe.substack.com/p/relentless-vibe-coding-part.... Even started building a little library to prove out the concept: https://github.com/Mockapapella/containment-chamber

Spoiler: there won't be a part 2, or if there is it will be with a different approach. I wrote a followup that summarizes my experiences trying this out in the real world on larger codebases: https://thelisowe.substack.com/p/reflections-on-relentless-v...

tl;dr I use a version of it in my codebases now, but the combination of LLM reward hacking and the long tail of verfiers in a language (some of which don't even exist! Like accurately detecting dead code in Python (vulture et. al can't reliably do this) or valid signatures for property-based tests) make this problem more complicated than it seems on the surface. It's not intractable, but you'd be writing many different language-specific libraries. And even then, with all of those verifiers in place, there's no guarantee that when working in different sized repos it will produce a consistent quality of code.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#190

Post by __MatrixMan__ »

jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

Isn't that true of everything else also? Facts about real things are the result of sampling reality several times and coming up with consistent stores about those things. The accuracy of those stories is always bounded by probabilities related to how complete your sampling strategy is.
Post reply on HN