Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

91–100 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#91

Post by HPsquared »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

There are plenty of LLM services that have a conversational style. The paragraph blocks thing is just a style.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#92

Post by keiferski »

Archelaos wrote: post #82
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

I never expected LLMs to be like an actual conversation between humans. The model is in some respects more capable and in some respects more limited than a human. I mean, one could strive for an exact replica of a human -- but for what purpose? The whole thing is a huge association machine. It is a surealistic inspiration generator for me. This is how it works at the moment, until the next break through ...

The disconnect is that companies are trying desperately to frame LLMs as actual entities and not just an inert tech tool. AGI as a concept is the biggest example of this, and the constant push to "achieve AGI" is what's driving a lot of stock prices and investment.

A strictly machinelike tool doesn't begin answers by saying "Great question!"

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#93

Post by HarHarVeryFunny »

psychoslave wrote: post #81
AlecSchueler wrote: post #64

Earlier quoted context omitted.

In a way though those things aren't so different as they might first appear. The factual answer is traditionally the most plausible response to many questions. They don't operate on any level other than pure language but there are a heap of behaviours which emerge from that.

Most plausible world model is not something stored raw in utterances. What we interpret from sentences is vastly different from what is extractable from mere sentences on their own. Facts, unlike fabulations, require crossing experience beyond the expressions on trial.

Right, facts need to be grounded and obtained from reliable sources such as personal experience, or a textbook. Just because statistically most people on Reddit or 4Chan said the moon is made of cheese doesn't make it so.

But again, LLMs don't even deal in facts, nor store any memories of where training samples came from, and of course have zero personal experience. It's just "he said, she said" put into a training sample blender and served one word at a time.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#94

Post by throw4847285 »

anal_reactor wrote: post #83
jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

This is exactly why I don't like dealing with most people.

Every thread like this I like to go through and count how many people are making the pro-AI "Argument from Misanthropy." Based on this exercise, I believe that the biggest AI boosters are simply the most disagreeable people in the industry, temperamentally speaking.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#95

Post by psychoslave »

actionfromafar wrote: post #69
rthrfrd wrote: post #31

Earlier quoted context omitted.

But we also have a stake in our society, in the form of a reputation or accountability, that greatly influences our behaviour. So comparing us to an LLM has always been meaningless anyway.

Hm, great lumps of money also detaches a person from reputation or accountability.

Money, or any single metrics, no matter how high, is not enough to bend someone actions in territory they will assess unacceptable otherwise.

How much money would make anyone accept to engage in a genocide by direct bribe? The thing is, some people would not see any amount as a convincing one, while some other will do it proactively for no money at all.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#96

Post by psychoslave »

pixl97 wrote: post #36
some_furry wrote: post #30

Earlier quoted context omitted.

Human minds are more complicated than a language model that behaves like a stochastic echo.

Birds are more complicated than jet engines, but jet engines travel a lot faster.

Jet engines don't go anywhere without a large industry continuously taking care of all the complexity that even the simplest jet travel imply.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#97

Post by tovej »

wisty wrote: post #76
HarHarVeryFunny wrote: post #68

Earlier quoted context omitted.

> I think they are much smarter than that. Or will be soon. It's not a matter of how smart they are (or appear), or how much smarter they may become - this is just the fundamental nature of Transformer-based LLMs and how they are trained. The sycophantic personality is mostly unrelated to this. Maybe it's part human preference (conferred via RLHF training), but the "You're asbolutely right! (I was wrong)" is clearly…

I'm not sure what you mean by "deals in facts, not words" means. Llm deal in vectors internally, not words. They explode the word into a multidimensional representation, and collapse it again, and apply the attention thingy to link these vectors together. It's not just a simple n:n Markov chain, a lot is happening under the hood. And are you saying the syncophant behaviour was deliberately programmed, or emerged beca…

If you're not sure, maybe you should look up the term "expert system"?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#98

Post by motoboi »

keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

Reflect a moment over the fact that LLMs currently are just text generators.

Also that the conversational behavior we see it’s just examples of conversations that we have the model to mimic so when we say “System: you are a helpful assistant. User: let’s talk. Assistant:” it will complete the text in a way that mimics a conversation?.

Yeah, we improved over that using reinforcement learning to steer the text generation into paths that lead to problem solving and more “agentic” traces (“I need to open this file the user talked about to read it and then I should run bash grep over it to find the function the user cited”), but that’s just a clever way we found to let the model itself discover which text generation paths we like the most (or are more useful to us).

So to comment on your discomfort, we (humans) trained the model to spill out answers (there are thousand of human being right now writing nicely though and formatted answers to common questions so that we can train the models on that).

If we try to train the models to mimic long dances into shared meaning we will probably decrease their utility. And we won’t be able anyway to do that because then we would have to have customized text traces for each individual instead of question-answers pairs.

Downvoters: I simplified things a lot here, in name of understanding, so bear with me.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#99

Post by flir »

CuriouslyC wrote: post #57
fzeindl wrote: post #20

Earlier quoted context omitted.

Bruce Schneier put it well: "Willison’s insight was that this isn’t just a filtering problem; it’s architectural. There is no privilege separation, and there is no separation between the data and control paths. The very mechanism that makes modern AI powerful - treating all inputs uniformly - is what makes it vulnerable. The security challenges we face today are structural consequences of using AI for everything." -…

Attributing that to Simon when people have been writing articles about that for the last year and a half doesn't seem fair. Simon gave that view visibility, because he's got a pulpit.

Longer, surely? (Though I don't have any evidence I can point to).

It's in-band signalling. Same problem DTMF, SS5, etc. had. I would have expected the issue to be intuitvely obvious to anyone who's heard of a blue box?

(LLMs are unreliable oracles. They don't need to be fixed, they need their outputs tested against reality. Call it "don't trust, verify").

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#100

Post by HarHarVeryFunny »

wisty wrote: post #76
HarHarVeryFunny wrote: post #68

Earlier quoted context omitted.

> I think they are much smarter than that. Or will be soon. It's not a matter of how smart they are (or appear), or how much smarter they may become - this is just the fundamental nature of Transformer-based LLMs and how they are trained. The sycophantic personality is mostly unrelated to this. Maybe it's part human preference (conferred via RLHF training), but the "You're asbolutely right! (I was wrong)" is clearly…

I'm not sure what you mean by "deals in facts, not words" means. Llm deal in vectors internally, not words. They explode the word into a multidimensional representation, and collapse it again, and apply the attention thingy to link these vectors together. It's not just a simple n:n Markov chain, a lot is happening under the hood. And are you saying the syncophant behaviour was deliberately programmed, or emerged beca…

LLMs are not like an expert system representing facts as some sort of ontological graph. What's happening under the hood is just whatever (and no more) was needed to minimize errors on it's word-based training loss.

I assume the sycophantic behavior is part because it "did well" during RLHF (human preference) training, and part deliberately encouraged (by training and/or prompting) as someone's judgement call of the way to best make the user happy and own up to being wrong ("You're absolutely right!").

Post reply on HN