Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

311–320 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#311

Post by janalsncm »

steer_dev wrote: post #3

OP here. I wrote this because I got tired of agents confidently guessing answers when they should have asked for clarification (e.g. guessing "Springfield, IL" instead of asking "Which state?" when asked "weather in Springfield"). I built an open-source library to enforce these logic/safety rules outside the model loop: https://github.com/imtt-dev/steer

You should use structured output rather than checking and rechecking for valid json. It can’t solve all of your problems but it can enforce a schema on the output format.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#312

Post by chemotaxis »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

This is not necessarily a fundamental limitation. It's a consequence of a fine-tuning process where human raters decide how "good" an answer is. They're not rating the flow of the conversation, but looking at how complete / comprehensive the answer to a one-shot question looks like. This selects for walls of overconfident text.

Another thing the vendors are selecting for is safety / PR risk. If an LLM answers to a hobby chemistry question in a matter-of-factly way, that's a disastrous PR headline in the making. If they open with several paragraphs of disclaimers or just refuse to answer, that's a win.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#313

Post by thfuran »

heresie-dabord wrote: post #270
gmueckl wrote: post #206

Earlier quoted context omitted.

It's the fact that these are competent human language word salad generators that messes with human psychology.

My calculator produces accurate, verifiable results. My calculator is a great tool but it is not a mathematician. Not by a long shot.

And you can tell this pretty easily because it can't have a conversation with you.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#314

Post by jandrese »

rockskon wrote: post #274
not2b wrote: post #267

Earlier quoted context omitted.

You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.

Do you honestly think an executive wanted a chat bot that confidently lies?

Do the lies look really good in a demo when you're pitching it to investors? Are they obscure enough that they aren't going to stand out? If so no problem.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#315

Post by seanmcdirmid »

mrguyorama wrote: post #302
OkayPhysicist wrote: post #244

Earlier quoted context omitted.

I suspect a lot of the em-dash usage also comes from transcriptions of verbal media. In the spoken word, people use the kinds of asides that elicit an em-dash a lot.

I would bet like a dollar that the supposed em-dash usage (which I'm not convinced is an accurate take in the first place) would have come from an enterprising dev somewhere being like "Well, we probably don't need multiple tokens for hyphens" and coercing every dash type thing to just one hyphen like token. But I'm also showing off my ignorance with how these machines turn text into tokens in practice.

Two dashes is still a token. You would only be correct if LLMs were still thinking at the level of characters.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#317

Post by Forgeties79 »

HarHarVeryFunny wrote: post #122
Forgeties79 wrote: post #59

Earlier quoted context omitted.

> You can't blame an LLM for getting the facts wrong, or hallucinating, when by design they don't even attempt to store facts in the first place On one level I agree, but I do feel it’s also right to blame the LLM/company for that when the goal is to replace my search engine of choice (my major tool for finding facts and answering general questions), which is a huge pillar of how they’re sold to/used by the public.

True, although that's a tough call for a company like Google. Even before LLMs people were asking Google search questions rather than looking for keyword matches, and now coupled with ChatGPT it's not surprising that people are asking the computer to answer questions and seeing this as a replacement for search. I've got to wonder how the typical non-techie user internalizes the difference between asking questions of…

Yeah I pretty much agree with everything you’ve got here

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#318

Post by ActorNightly »

vidarh wrote: post #269
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

A lot of this, I suspect, on the basis of having worked on a supervised fine-tuning project for one of the largest companies in this space, is that providers have invested a lot of money in fine-tuning datasets that sound this way. On the project I did work on, reviewers were not allowed to e.g. answer that they didn't know - they had to provide an answer to every prompt provided. And so when auditing responses, a lo…

Its not the training/tuning, its pretty much the nature of llms. The whole idea is to give a best quess of the token. The more complex dynamics behind the meaning of the words and how those words relate to real world concepts isn't learned.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#319

Post by andoando »

ryandrake wrote: post #161
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

There is sort of the opposite problem as well, as the top comment was saying, where it can super confidently propose that its absolutely right and you're wrong instead of asking questions to try and understand what you mean.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#320

Post by ares623 »

ryandrake wrote: post #161
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…

Looking forward to living in a society where everyone feels like they’re CEOs.
Post reply on HN