Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

251–260 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#251

Post by mannanj »

ryandrake wrote: post #161
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

Isn’t it kind of true that the systems we as servile people-pleasers have to operate out of are exactly these? The hierarchical status games and alpha-animal tribal dynamics are these. Our leaders who are so might and rich and powerful want to keep their position, and we don’t want to admit they have more influence than we do for things like AI now and so we stand and watch naively as they reward the people pleasers and eventually historically we learn(ed) it pays to please until leadership changes.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#252

Post by HarHarVeryFunny »

coldtea wrote: post #227
HarHarVeryFunny wrote: post #166

Earlier quoted context omitted.

Animal/human brains and LLMs have fundamentally different goals (or loss functions, if you prefer), even though both are based around prediction. LLMs are trained to auto-regressively predict text continuations. They are not concerned with the external world and any objective experimentally verifiable facts - they are just self-predicting "this is what I'm going to say next", having learnt that from the training data…

> Humans/animals are embodied, living in the real world, whose design has been honed by a "loss function" favoring survival. Animals are "designed" to learn facts about the real world, and react to those facts in a way that helps them survive. Yes - but LLMs also get this "embodied knowledge" passed down from human-generated training data. We are their sensory inputs in a way (which includes their training images, au…

> Yes - but LLMs also get this "embodied knowledge" passed down from human-generated training data.

It's not the same though. It's the difference between reading about something and, maybe having read the book and/or watched the video, learning to DO it yourself, acting based on the content of your own mind.

The LLM learns 2nd hand heresay, with no idea of what's true or false, what generalizations are valid, or what would be hallucinatory, etc, etc.

The human learns verifiable facts, uses curiosity to explore and fill the gaps, be creative etc.

I think it's pretty obvious why LLMs have all the limitations and deficiencies that they do.

If 2nd hand heresay (from 1000's of conflicting sources) really was good as 1st hand experience and real-world prediction, then we'd not be having this discussion - we'd be bowing to our AGI overlords (well, at least once the AI also got real-time incremental learning, internal memory, looping, some type of (virtual?) embodiment, autonomy ...).

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#253

Post by wordpad »

Mockapapella wrote: post #189

I wrote about something like this a couple months ago: https://thelisowe.substack.com/p/relentless-vibe-coding-part... . Even started building a little library to prove out the concept: https://github.com/Mockapapella/containment-chamber Spoiler: there won't be a part 2, or if there is it will be with a different approach. I wrote a followup that summarizes my experiences trying this out in the real world on larger c…

How are vibe coding platforms solving this?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#254

Post by _yb2s »

psychoslave wrote: post #86
_yb2s wrote: post #75

Earlier quoted context omitted.

Specifically, they are capable of inductive logic but not deductive logic. In practice, this may not be a serious limitation, if they get good enough at induction to still almost always get the right answer.

What about abduction though?

You’ll have to wait for the FOOM “Fast Onset of Overwhelming Mastery” for that I’m afraid.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#255

Post by bsshjdjddjdj »

encyclopedism wrote: post #218
jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

I couldn't agree with you more. I really do find it puzzling so many on HN are convinced LLM's reason or think and continue to entertain this line of reasoning. At the same time also somehow knowing what precisely the brain/mind does and constantly using CS language to provide correspondences where there are none. The simplest example being that LLM's somehow function in a similar fashion to human brains. They catego…

People believe that because they are financially invested in it. Everyone has known LLMs are bullshit for years now.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#256

Post by LogicFailsMe »

pessimizer wrote: post #248
LogicFailsMe wrote: post #212

Earlier quoted context omitted.

Nearly every woman I know who is an English as a second language speaker is leaning hard into these things currently to make their prose sound more natural. And that has segued into them being treated almost as a confidant or a friend. As flawed as they are currently, I remain astounded that people think they will never improve and that people don't want a plastic pal who's fun to be with(tm). I find them frustrating…

> I get science fiction in return. And once this garbage is in your context, it's polluting everything that comes after. If they don't know, I need them to shut up. But they don't know when they don't know. They don't know shit.

I am reminded of AI summaries and Microsoft Copilot. All push low value. But I separate that from the underlying potential of the technology. And I wish we heard more from deep domain experts like Karpathy and less from influencer dilettantes like Dylan Patel about where this is going.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#257

Post by jacquesm »

lubujackson wrote: post #250
jacquesm wrote: post #140

LLMs are text model, not world models and that is the root cause of the problem. If you and I would be discussing furniture and for some reason you had assumed the furniture to be glued to the ceiling instead of standing on the floor (contrived example) then it would most likely only take one correction based on your actual experience that you are probably on the wrong track. An LLM will happily re-introduce that err…

The "world model" is what we often refer to as the "context". But it is hard to anticipate bad assumptions that seem obvious because of our existing world model. One of the first bugs I scanned past from LLM generated code was something like: if user.id == "id": ... Not anticipating that it would arbitrarily put quotes around a variable name. Other time it will do all kinds of smart logic, generate data with ids then…

> The "world model" is what we often refer to as the "context".

No, we often do not, and when we do that's just plain wrong.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#258

Post by bwahah4 »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

In the US anyway, most adults read at a middle school level.

It's not "masquerading as a human". The majority of humans are functional illiterates who only understand the world through the elementary principles of their local culture.

It's the minority of the human species that take what amounts to little more than arguing semantics that need the reality check. Unless one is involved in work that directly impacts public safety (defined as harm to biology) the demand to apply one concept or another is arbitrary preference.

Healthcare, infrastructure, and essential biological support services are all most humans care about. Everything else the majority see as academic wank.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#259

Post by rglover »

I think this is for the best. Let the "confident idiot" types briefly entertain the idea of competency, hit the inevitable wall, and go away for good. It will take a few years, lots of mistakes, and billions (if not trillions) wasted, but those people will drift back to the mean or lower when they realize ChatGPT isn't the ghost of Robin Leach.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#260

Post by __turbobrew__ »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

The day when the LLM responds to my question with another question will be quite interesting. Especially at work, when someone asks me a question I need to ask for clarifying information to answer the original question fully.
Post reply on HN