Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

371–380 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#371

Post by [deleted] »

anal_reactor wrote: post #364
throw4847285 wrote: post #285

Earlier quoted context omitted.

The problem is one of negative polarization. I found myself skeptical of a lot of the claims around LLMs, but was annoyed by AI critics forming an angry mob anytime AI was used for anything. However, I still considered myself in that camp, and ended up far more annoyed by AI boosterism than AI skepticism, which pushed me in the direction of being even more negative about AI than I started. It's the mirror of what hap…

It sounds silly to me not because I don't value humans. I don't value humans because of my personal grievances that are difficult to defend in a serious ethical discussion. It sounds silly to me because it leaves "human" undefined. To me, the question "is LLM human?" is eerily similar to "are black people people?" and "are Jews people?". AI displays intelligence but it doesn't deserve respect because it doesn't meet…

[deleted]

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#372

Post by dontlikeyoueith »

rockskon wrote: post #299
dontlikeyoueith wrote: post #287

Earlier quoted context omitted.

In practice, yes, though they wouldn't think of it that way because that's the kind of people they surround themselves with, so it's what they think human interaction is actually like.

"I want a chat bot that's just as reliable at Steve! Sure he doesn't get it right all the time and he cost us the Black+Decker contract, but he's so confident!" You're right! This is exactly what an executive wants to base the future of their business off of!

Yes, that is in fact their revealed preference.

Did you have a point?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#373

Post by jacquesm »

monkpit wrote: post #365
jacquesm wrote: post #325

Earlier quoted context omitted.

What's next, motivational speaking for LLMs?

I remember reading about speaking in an encouraging manner to agentic AI leading to better results, but I can’t seem to find a citation for this.

That's pathetic. Pleading comes next then. And after that most likely praying.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#375

Post by rockskon »

dontlikeyoueith wrote: post #372
rockskon wrote: post #299

Earlier quoted context omitted.

"I want a chat bot that's just as reliable at Steve! Sure he doesn't get it right all the time and he cost us the Black+Decker contract, but he's so confident!" You're right! This is exactly what an executive wants to base the future of their business off of!

Yes, that is in fact their revealed preference. Did you have a point?

You use unfalsifiable logic. And you seem to argue that, given the choice, CEOs would prefer not to maximize revenue in favor of... what, affection for an imaginary intern?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#376

Post by __del__ »

lkbm wrote: post #219
HPsquared wrote: post #165

Earlier quoted context omitted.

I usually do the "drip feed" with ChatGPT, but maybe that's not optimal. Hmm, maybe info dump is a good thing to try.

There a recent(ish: May 2025) paper about how drip-feeding information is worse than restarting with a revised prompt once you realize details are missing.[0] [0] https://arxiv.org/abs/2505.06120

this has been my casual finding as well. why would i want all that conversational crap in the context window?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#377

Post by zenoprax »

danuker wrote: post #265
zenoprax wrote: post #146

Earlier quoted context omitted.

ChatGPT offered a "robotic" personality which really improved my experience. My frustrations were basically decimated right away and I quickly switched to a more "You get out of it what you put in" mindset. And less than two weeks in they removed it and replaced it with some sort of "plain and clear" personality which is human-like. And my frustrations ramped up again. That brief experiment taught me two things: 1. I…

> I should be running my own LLM I approve of this, but in your place I'd wait for hardware to become cheaper when the bubble blows over. I have a i9-10900, and bought an M.2 SSD and 64GB of RAM in july for it, and get useful results with Qwen3-30B-A3B (some 4-bit quant from unsloth running on llama.cpp). It's much slower than an online service (~5-10 t/s), and lower quality, but it still offers me value for my use c…

I have a 9070 XT (16 GB VRAM) and it is fast with deepseek-r1:14B but I didn't know about that Qwen model. Most of the 'better' models will crash for lack of RAM.

https://dev.to/composiodev/qwen-3-vs-deep-seek-r1-evaluation...

If it runs, it looks like I can get a bit more quality. Thanks for the suggestion.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#378

Post by anal_reactor »

throw4847285 wrote: post #370
anal_reactor wrote: post #364

Earlier quoted context omitted.

It sounds silly to me not because I don't value humans. I don't value humans because of my personal grievances that are difficult to defend in a serious ethical discussion. It sounds silly to me because it leaves "human" undefined. To me, the question "is LLM human?" is eerily similar to "are black people people?" and "are Jews people?". AI displays intelligence but it doesn't deserve respect because it doesn't meet…

Now I understand your love of LLMs. What you write reads like the output of an LLM but with the dial turned from obsequious to edgelord. There is no content, just posturing. None of what you wrote holds up to any scrutiny, and much of it is internally contradictory, but it doesn't really matter to you, I guess. I don't think you're even talking to me.

I take it as a compliment. I've always been like this. I challenged core assumptions, people didn't like it, later it would turn out I was right.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#379

Post by tovej »

wisty wrote: post #351
tovej wrote: post #97

Earlier quoted context omitted.

If you're not sure, maybe you should look up the term "expert system"?

It was a polite way of saying "that's kinda bull". And yes, I know what an expert system is. Do you know that a neural network (or set of matrices, same thing really) can approximate anything else? https://en.wikipedia.org/wiki/Universal_approximation_theore... How do you know that inside the black box, they don't approximate expert systems?

I'm not sure you do, because expert systems are constraint solvers and LLMs are not. They literally deal in encoded facts, which is what the original comment was about.

The universal approximation theorem is not relevant. You would first have to try to train the neural network to approximate a constraint solver (that's not the case with LLMs), and in practice, these kinds of systems are exactly the ones that a neural network is bad at.

The universal approximation theory says nothing about feasibility, it only talks about theoretical existence as a mathematical object, not whether the object can actually be created in the real world.

I'll remind you that the expert system would have to have been created and updated by humans. It would have had to have been created before a neural network was applied to it in the first place.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#380

Post by ako »

jacquesm wrote: post #288
ryandrake wrote: post #161

Earlier quoted context omitted.

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

> "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly point it out, and then all sorry-this and apologize-that and "here's the actual answer!" It's exactly the kind of personality you always see bubbling up into the orbit of a rich and powerful tech CEO. You ma…

Maybe it’s just the fact that many models are trained by americans? I’ve seen great improvements in answers by asking it to “tone it down, answer like you’re British”.
Post reply on HN