Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

291–300 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#291

Post by jacquesm »

Cheer2171 wrote: post #241
ryandrake wrote: post #161

Earlier quoted context omitted.

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

> LLMs all Sounds like you don't know how RLHF works. Everything you describe is post-training. Base models can't even chat, they have to be trained to even do basic conversational turn taking.

> Everything you describe is post-training. Base models can't even chat, they have to be trained to even do basic conversational turn taking.

So, that's still training then, so not 'post-training'. Just a different training phase.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#292

Post by MangoToupe »

vlowther wrote: post #185
MangoToupe wrote: post #109

Earlier quoted context omitted.

> Reflect a moment over the fact that LLMs currently are just text generators. You could say the same thing about humans.

No, you cannot. Our abstract language abilities (especially the written word part) are a very thin layer on top of hundreds of millions of years of evolution in an information dense environment.

Sure, but language is the only thing that meaningfully separates us from other great apes

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#293

Post by emp17344 »

MangoToupe wrote: post #109
motoboi wrote: post #98

Earlier quoted context omitted.

Reflect a moment over the fact that LLMs currently are just text generators. Also that the conversational behavior we see it’s just examples of conversations that we have the model to mimic so when we say “System: you are a helpful assistant. User: let’s talk. Assistant:” it will complete the text in a way that mimics a conversation?. Yeah, we improved over that using reinforcement learning to steer the text generati…

> Reflect a moment over the fact that LLMs currently are just text generators. You could say the same thing about humans.

How do you reconcile this belief with the fact that we evolved from organisms that had no concept of text?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#294

Post by jacquesm »

AznHisoka wrote: post #157
goalieca wrote: post #149

Earlier quoted context omitted.

This is a recent phenomenon. It seems most of the pages today are SEO optimized LLM garbage with the aim of having you scroll past three pages of ads. THe internet really used to be efficient and i could always find exactly what i wanted with an imprecise google search ~ 15 years ago.

Don’t you get this today with AI Overviews summarizing everything on top of most Google results?

Those AI overviews are dumb and wrong so often I have cut them out of the results entirely. They're embarrassing, really.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#295

Post by MangoToupe »

nosianu wrote: post #156
MangoToupe wrote: post #109

Earlier quoted context omitted.

> Reflect a moment over the fact that LLMs currently are just text generators. You could say the same thing about humans.

The human world model is based on physical sensors and actions. LLMs are based on our formal text communication. Very different! Just yesterday I observed myself acting on an external stimulus without any internal words (this happens continuously, but it is hard to notice because we usually don't pay attention to how we do things): I sat in a waiting area of a cinema. A woman walked by and dropped her scarf without n…

Without language we're just bald, bipedal chimps. Language is what makes us human.

> The human world model

Bruh this concept is insane

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#296

Post by _alternator_ »

This comment will probably get buried because I’m late to the party, but I’d like to point out that while they identify a real problem, the author’s approach—using code or ASTs to validate LLM output—does not solve it.

Yes, the approach can certainly detect (some) LLM errors, but it does not provide a feasible method to generate responses that don’t have the errors. You can see at the end that the proposed solution is to automatically update the prompt with a new rule, which is precisely the kind of “vibe check” that LLMs frequently ignore. If they didn’t, you could just write a prompt that says “don’t make any mistakes” and be done with it.

You can certainly use this approach to do some RL on LLM code output, but it’s not going to guarantee correctness. The core problem is that LLMs do next-token prediction and it’s extremely challenging to enforce complex rules like “generate valid code” a priori.

As a closing comment, it seems like I’m seeing a lot of technical half-baked stuff related to LLMs these days because LLMs are good at supporting people when they have half baked ideas, and are reluctant to openly point out the obvious flaws.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#297

Post by antonvs »

amorroxic wrote: post #34
steer_dev wrote: post #3

OP here. I wrote this because I got tired of agents confidently guessing answers when they should have asked for clarification (e.g. guessing "Springfield, IL" instead of asking "Which state?" when asked "weather in Springfield"). I built an open-source library to enforce these logic/safety rules outside the model loop: https://github.com/imtt-dev/steer

Thanks a lot for this. Also one question in case anyone could shed a bit of light: my understanding is that setting temperature=0, top_p=1 would cause deterministic output (identical output given identical input). For sure it won’t prevent factually wrong replies/hallucination, only maintains generation consistency (eq. classification tasks). Is this universally correct or is it dependent on model used? (or downright…

> my understanding is that setting temperature=0, top_p=1 would cause deterministic output (identical output given identical input).

That's typically correct. Many models are implemented this way deliberately. I believe it's true of most or all of the major models.

> Is this universally correct or is it dependent on model used?

There are implementation details that lead to uncontrollable non-determinism if they're not prevented within the model implementation. See e.g. the Pytorch docs for CUDA convolution determinism: https://docs.pytorch.org/docs/stable/notes/randomness.html#c...

That documents settings like this:

    torch.backends.cudnn.deterministic = True
Parallelism can be a source of non-determinism if it's not controlled for, either implicitly via e.g. dependencies or explicitly.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#298

Post by mrguyorama »

rockskon wrote: post #274
not2b wrote: post #267

Earlier quoted context omitted.

You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.

Do you honestly think an executive wanted a chat bot that confidently lies?

People with immense wealth, connections, influence, and power demonstrably struggle to not surround themselves with people who only say what the powerful person already wants to hear regardless of reality.

Putin didn't think Russia could take Ukraine in 3 days with literal celebration by the populace because he only works with honest folks for example.

Rich people get disconnected from reality because people who insist on speaking truth and reality around them tend to stop getting invited to the influence peddling sessions.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#299

Post by rockskon »

dontlikeyoueith wrote: post #287
rockskon wrote: post #274

Earlier quoted context omitted.

Do you honestly think an executive wanted a chat bot that confidently lies?

In practice, yes, though they wouldn't think of it that way because that's the kind of people they surround themselves with, so it's what they think human interaction is actually like.

"I want a chat bot that's just as reliable at Steve! Sure he doesn't get it right all the time and he cost us the Black+Decker contract, but he's so confident!"

You're right! This is exactly what an executive wants to base the future of their business off of!

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#300

Post by [deleted] »

not2b wrote: post #267
rockskon wrote: post #233

Earlier quoted context omitted.

Analogies of LLMs to humans obfuscates the problem. LLMs aren't like humans of any sort in any context. They're chat bots. They do not "think" like humans and applying human-like logic to them does not work.

You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.

[deleted]
Post reply on HN