Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

201–210 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#201

Post by i80and »

AznHisoka wrote: post #157
goalieca wrote: post #149

Earlier quoted context omitted.

This is a recent phenomenon. It seems most of the pages today are SEO optimized LLM garbage with the aim of having you scroll past three pages of ads. THe internet really used to be efficient and i could always find exactly what i wanted with an imprecise google search ~ 15 years ago.

Don’t you get this today with AI Overviews summarizing everything on top of most Google results?

The AI Overviews are... extremely bad. For most of my queries, Google's AI Overview misrepresents its own citations, or almost as bad, confidently asserts a falsehood or half-truth based on results that don't actually contain an answer to my search query.

I had the same issue with Kagi, where I'd follow the citation and it would say the opposite of the summary.

A human can make sense of search results with a little time and effort, but current AI models don't seem to be able to.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#203

Post by rkj93 »

falcor84 wrote: post #129
wongarsu wrote: post #102

Earlier quoted context omitted.

> but for what purpose? I recently introduced a non-technical person to Claude Code, and this non-human behavior was a big sticking point. They tried to talk to Claude similar as to a human, presenting it one piece of information at a time. With humans this is generally beneficial, and they will either nod for you to continue or ask clarifying questions. With Claude this does not work well, you have to infodump as mu…

I often find myself in these situations where I'm afraid that if I don't finish infodumping everything in a single message, it'll go in the wrong direction. So what I've been doing is switching it back to Plan Mode (even when I don't need a plan as such), just as a way of telling it "Hold still, we're still having a conversation".

I do this with cursor ai too. I tell, don't change anything, let me hear out what you plan to fix and what you will change

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#204

Post by ses1984 »

AznHisoka wrote: post #157
goalieca wrote: post #149

Earlier quoted context omitted.

This is a recent phenomenon. It seems most of the pages today are SEO optimized LLM garbage with the aim of having you scroll past three pages of ads. THe internet really used to be efficient and i could always find exactly what i wanted with an imprecise google search ~ 15 years ago.

Don’t you get this today with AI Overviews summarizing everything on top of most Google results?

It’s fine about 80% of the time, but the other 20% is a lot harder to answer because of lower quality results.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#205

Post by dominotw »

toddmorey wrote: post #42

Confident idiot: I’m exploring using LLM for diagram creation. I’ve found after about 3 prompts to edit an image with Gemini, it will respond randomly with an entirely new image. Another quirk is it will respond “here’s the image with those edits” with no edits made. It’s like a toaster that will catch on fire every eighth or ninth time. I am not sure how to mitigate this behavior. I think maybe an LLM as a judge ste…

same. i gave it a very well hand drawn floor plan but never seems to be able to create a formal version of it. Its very very simple too.

makes hilarious mistakes like putting toilet right in the middle of living room.

I dont get all the hype. am i stupid.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#206

Post by gmueckl »

heresie-dabord wrote: post #176
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

> The thing that bothers me the most about LLMs is What bothers me the most is the seemingly unshakable tendency of many people to anthropomorphise this class of software tool as though it is in any way capable of being human . What is it going to take? Actual, significant loss of life in a medical (or worse, military) context?

It's the fact that these are competent human language word salad generators that messes with human psychology.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#207

Post by dominotw »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

same experience. i try to learn with it but i can't really tell if what its teaching me is actually correct or merely making up when i challenge it with followup questions.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#208

Post by chrischen »

antonvs wrote: post #147
chrischen wrote: post #60

We already have verification layers: high level strictly typed languages like Haskell, Ocaml, Rescript/Melange (js ecosystem), purescript (js), elm, gleam (erlang), f# (for .net ecosystem). These aren’t just strict type systems but the language allows for algebraic data types, nominal types, etc, which allow for encoding higher level types enforced by the language compiler. The AI essentially becomes a glorified blan…

Interestingly, coding models often struggle with complex type systems, e.g. in Haskell or Rust. Of course, part of this has to do with the relative paucity of relevant training data, but there are also "cognitive" factors that mirror what humans tend to struggle with in those languages. One big factor behind this is the fact that you're no longer just writing programs and debugging them incrementally, iteratively dea…

Actually I found the coding models to work really well with these languages. And the type systems are not actually complex. Ocaml's type system is actually really simple, which is probably why the compiler can be so fast. Even back in the "beta" days of Copilot, despite being marketed as Python only, I found it worked for Ocaml syntax and worked just as well.

The coding models work really well with esoteric syntaxes so if the biggest hurdle to adoption of haskell was syntax, that's definitely less of a hurdle now.

> Instead, you're writing non-trivial proofs about all possible runs of the program.

All possible runs of a program is exactly what HM type systems type check for. This fed into the coding model automatically iterates until it finds a solution that doesn't violate any possible run of the program.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#209

Post by jordanb »

rzwitserloot wrote: post #171
ryandrake wrote: post #161

Earlier quoted context omitted.

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

I don't think these LLMs were explicitly designed based on the CEO's detailed input that boils down to 'reproduce these servile yes-men in LLM form please'. Which makes it more interesting. Apparently reddit was a particularly hefty source for most LLMs; your average reddit conversation is absolutely nothing like this. Separate observation: That kind of semi-slimey obsequious behaviour annoys me. Significantly so. It…

The servile stuff was trained into them with RLHF with the trainers largely being low-wage workers in the global south. That's also where some of the other stuff like excessive em-dash stuff came from. I think it's a combination of those workers anticipating how they would be expected to respond by a first-world employer, and also explicit instructions given to them about how the robot should be trained.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#210

Post by LogicFailsMe »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

My favorite description of an LLM so far is of a typical 37-year-old male Reddit user. And in that sense, we have already created the AGI.
Post reply on HN