Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

271–280 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#271

Post by Retric »

rockskon wrote: post #233
ryandrake wrote: post #161

Earlier quoted context omitted.

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

Analogies of LLMs to humans obfuscates the problem. LLMs aren't like humans of any sort in any context. They're chat bots. They do not "think" like humans and applying human-like logic to them does not work.

It’s not about thinking, it’s about what they are trained to do. You could train a LLM to always respond to every prompt by repeating the prompt in Spanish, but that’s not the desired behavior.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#272

Post by codazoda »

toddmorey wrote: post #42

Confident idiot: I’m exploring using LLM for diagram creation. I’ve found after about 3 prompts to edit an image with Gemini, it will respond randomly with an entirely new image. Another quirk is it will respond “here’s the image with those edits” with no edits made. It’s like a toaster that will catch on fire every eighth or ninth time. I am not sure how to mitigate this behavior. I think maybe an LLM as a judge ste…

I had a similar result trying to create 16 similarly styled images. After half a dozen it just started kicking out the same image over and over again no matter what the prompt said. Even the “thinking” looked right, but the image was just a repeat. I don’t know if this is some type of context limitation or what.

I got around it by using a new prompt/context for each image. This required some rethinking about how to make them match. What I did was create a sprite sheet with the first prompt and then only replaced (edited) the second prompt.

I still got some consistency problems because there were a few important details left out of my sprite sheet. Next time I think I’ll create those individually and then attach them as context for additional prompts.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#273

Post by bwahah4 »

pessimizer wrote: post #246
pardon_me wrote: post #173

Earlier quoted context omitted.

The problem with these LLM chat-bots is they are too human, like a mirror held up to the plastic-fantastic society we have morphed into. Naturally programmed to serve as a slave to authority, this type of fake conversation is what we've come to expect as standard. Big smiles everyone! Big smiles!!

Nah. Talking like an LLM would get you fired in a day. People are already suspicious of ass-kissers, they hate it when they think people are not listening to them, and if you're an ass-kisser who's not listening and is then wrong about everything, they want you escorted out by security. The real human position would be to be an ass-kisser who hangs on every word you say, asks flattering questions to keep you talking,…

[flagged]

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#274

Post by rockskon »

not2b wrote: post #267
rockskon wrote: post #233

Earlier quoted context omitted.

Analogies of LLMs to humans obfuscates the problem. LLMs aren't like humans of any sort in any context. They're chat bots. They do not "think" like humans and applying human-like logic to them does not work.

You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.

Do you honestly think an executive wanted a chat bot that confidently lies?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#275

Post by [deleted] »

falcor84 wrote: post #129
wongarsu wrote: post #102

Earlier quoted context omitted.

> but for what purpose? I recently introduced a non-technical person to Claude Code, and this non-human behavior was a big sticking point. They tried to talk to Claude similar as to a human, presenting it one piece of information at a time. With humans this is generally beneficial, and they will either nod for you to continue or ask clarifying questions. With Claude this does not work well, you have to infodump as mu…

I often find myself in these situations where I'm afraid that if I don't finish infodumping everything in a single message, it'll go in the wrong direction. So what I've been doing is switching it back to Plan Mode (even when I don't need a plan as such), just as a way of telling it "Hold still, we're still having a conversation".

[deleted]

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#276

Post by ux266478 »

heresie-dabord wrote: post #176
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

> The thing that bothers me the most about LLMs is What bothers me the most is the seemingly unshakable tendency of many people to anthropomorphise this class of software tool as though it is in any way capable of being human . What is it going to take? Actual, significant loss of life in a medical (or worse, military) context?

That qualifier only makes the anthropormorphization more sound. Have you actually thought it through? Give an untrained and unspecialized human the power to cause significant loss of life in a medical context in the same exact capacity, and it's all but guaranteed that's the outcome you'll end up with.

I think it's important to be skeptical and push back against a lot of the ridiculous mass-adoption of LLMs, but not if you can't actually make a well-reasoned point. I don't think you realize the damage you do when the people gunning for mass proliferation of LLMs in places they don't belong can only find examples of incoherent critique.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#277

Post by jacquesm »

pessimizer wrote: post #263
coldtea wrote: post #145

Earlier quoted context omitted.

> LLMs are text model, not world models and that is the root cause of the problem. Is it though? In the end, the information in the training texts is a distilled proxy for the world, and the weighted model ends up being a world model, just an once-removed one. Text is not that different to visual information in that regard (and humans base their world model on both). > Not having a world model is a massive disadvanta…

People have an actual world model, though, that they have to deal with in order to get the food into their mouths or to hit the toilet properly. The "facts" that they believe that may be nonsense are part of an abstract world model that is far from their experience, for which they never get proper feedback (such as the political situation in Bhutan, or how their best friend is feeling.) In those, it isn't surprising…

> Human abstractions are based in the reality of the physical responses of the people around them.

And in the physical responses of the world around them. That empiricism is the foundation of all of science and if you throw that out the end result is gibberish.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#278

Post by wat10000 »

i80and wrote: post #201
AznHisoka wrote: post #157

Earlier quoted context omitted.

Don’t you get this today with AI Overviews summarizing everything on top of most Google results?

The AI Overviews are... extremely bad . For most of my queries, Google's AI Overview misrepresents its own citations, or almost as bad, confidently asserts a falsehood or half-truth based on results that don't actually contain an answer to my search query. I had the same issue with Kagi, where I'd follow the citation and it would say the opposite of the summary. A human can make sense of search results with a little…

Cheap AI models aren't good at this, anyway, and AI Overviews have to use cheap models since they get used so much. They would be a lot better (still need to check, but they'd be much less stupid) if they used something like GPT-5, but that's just not feasible right now.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#279

Post by TheOtherHobbes »

HarHarVeryFunny wrote: post #68
wisty wrote: post #53

Earlier quoted context omitted.

I think they are much smarter than that. Or will be soon. But they are like a smart student trying to get a good grade (that's how they are trained!). They'll agree with us even if they think we're stupid, because that gets them better grades, and grades are all they care about. Even if they are (or become) smart enough to know better, they don't care about you. They do what they were trained to do. They are becoming…

> I think they are much smarter than that. Or will be soon. It's not a matter of how smart they are (or appear), or how much smarter they may become - this is just the fundamental nature of Transformer-based LLMs and how they are trained. The sycophantic personality is mostly unrelated to this. Maybe it's part human preference (conferred via RLHF training), but the "You're asbolutely right! (I was wrong)" is clearly…

It's worse than that. LLMs are slightly addictive because of intermittent reinforcement.

If they give you nonsense most of the time and an amazing answer occasionally you'll bond with them far more strongly than if they're perfectly correct all time.

Selective reinforcement means you get hooked more quickly if the slot machine pays out once every five times than if it pays out on each spin.

That includes "That didn't work because..." debugging loops.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#280

Post by zby »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

When I expect it to do that I just end my prompt with '. Discuss' - usually this works really well. Not exactly human like - it tries to list all questions and variants at once - but most with good default answers so I only need to engage with a couple of them.
Post reply on HN