Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

61–70 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#61

Post by rafamct »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

Yes you're totally right! I misunderstood what you meant, let me write six more paragraphs based on a similar misunderstanding rather than just trying to get clarification from you

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#62

Post by 6LLvveMx2koXfwn »

CuriouslyC wrote: post #57
fzeindl wrote: post #20

Earlier quoted context omitted.

Bruce Schneier put it well: "Willison’s insight was that this isn’t just a filtering problem; it’s architectural. There is no privilege separation, and there is no separation between the data and control paths. The very mechanism that makes modern AI powerful - treating all inputs uniformly - is what makes it vulnerable. The security challenges we face today are structural consequences of using AI for everything." -…

Attributing that to Simon when people have been writing articles about that for the last year and a half doesn't seem fair. Simon gave that view visibility, because he's got a pulpit.

He referenced Simon's article from September the 12th 2022

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#63

Post by ecocentrik »

kissgyorgy wrote: post #16

It's just simple validation with some error logging. Should be done the same way as for humans or any other input which goes into your system. LLM provides inputs to your system like any human would, so you have to validate it. Something like pydantic or Django forms are good for this.

I agree. Agentic use isn't always necessary. Most of the time it makes more sense to treat LLMs like a dumb, unauthenticated human user.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#64

Post by AlecSchueler »

HarHarVeryFunny wrote: post #24
jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

The factuality problem with LLMs isn't because they are non-deterministic or statistically based, but simply because they operate at the level of words, not facts. They are language models. You can't blame an LLM for getting the facts wrong, or hallucinating, when by design they don't even attempt to store facts in the first place. All they store are language statistics, boiling down to "with preceding context X, mos…

In a way though those things aren't so different as they might first appear. The factual answer is traditionally the most plausible response to many questions. They don't operate on any level other than pure language but there are a heap of behaviours which emerge from that.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#66

Post by TimPC »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

The benchmarks are dumb but highly followed so everyone optimizes for the wrong thing.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#67

Post by knollimar »

etamponi wrote: post #45

Aren't we just reinventing programming languages from the ground up? This is the loop (and honestly, I predicted it way before it started): 1) LLMs can generate code from "natural language" prompts! 2) Oh wait, I actually need to improve my prompt to get LLMs to follow my instructions... 3) Oh wait, no matter how good my prompt is, I need an agent (aka a for loop) that goes through a list of deterministic steps so th…

The hard part is just learning interfaces quickly for programming. If only we had a good tool for that.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#68

Post by HarHarVeryFunny »

wisty wrote: post #53
HarHarVeryFunny wrote: post #24

Earlier quoted context omitted.

The factuality problem with LLMs isn't because they are non-deterministic or statistically based, but simply because they operate at the level of words, not facts. They are language models. You can't blame an LLM for getting the facts wrong, or hallucinating, when by design they don't even attempt to store facts in the first place. All they store are language statistics, boiling down to "with preceding context X, mos…

I think they are much smarter than that. Or will be soon. But they are like a smart student trying to get a good grade (that's how they are trained!). They'll agree with us even if they think we're stupid, because that gets them better grades, and grades are all they care about. Even if they are (or become) smart enough to know better, they don't care about you. They do what they were trained to do. They are becoming…

> I think they are much smarter than that. Or will be soon.

It's not a matter of how smart they are (or appear), or how much smarter they may become - this is just the fundamental nature of Transformer-based LLMs and how they are trained.

The sycophantic personality is mostly unrelated to this. Maybe it's part human preference (conferred via RLHF training), but the "You're asbolutely right! (I was wrong)" is clearly deliberately trained, presumably as someone's idea of the best way to put lipstick on the pig.

You could imagine an expert system, CYC perhaps, that does deal in facts (not words) with a natural language interface, but still had a sycophantic personality just because someone thought it was a good idea.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#69

Post by actionfromafar »

rthrfrd wrote: post #31
Davidzheng wrote: post #10

Earlier quoted context omitted.

lol humans are non-deterministic too

But we also have a stake in our society, in the form of a reputation or accountability, that greatly influences our behaviour. So comparing us to an LLM has always been meaningless anyway.

Hm, great lumps of money also detaches a person from reputation or accountability.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#70

Post by someguy101010 »

wrote about this a bit too in https://www.robw.fyi/2025/10/24/simple-control-flow-for-auto...

ran into this when writing agents to fix unit tests. often times they would just give up early so i started writing the verifiers directly into the agent's control flow and this produced much more reliable results. i believe claude code has hooks that do something similar as well.

Post reply on HN