Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

351–360 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#351

Post by wisty »

tovej wrote: post #97
wisty wrote: post #76

Earlier quoted context omitted.

I'm not sure what you mean by "deals in facts, not words" means. Llm deal in vectors internally, not words. They explode the word into a multidimensional representation, and collapse it again, and apply the attention thingy to link these vectors together. It's not just a simple n:n Markov chain, a lot is happening under the hood. And are you saying the syncophant behaviour was deliberately programmed, or emerged beca…

If you're not sure, maybe you should look up the term "expert system"?

It was a polite way of saying "that's kinda bull".

And yes, I know what an expert system is.

Do you know that a neural network (or set of matrices, same thing really) can approximate anything else? https://en.wikipedia.org/wiki/Universal_approximation_theore...

How do you know that inside the black box, they don't approximate expert systems?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#352

Post by mrguyorama »

jacquesm wrote: post #277
pessimizer wrote: post #263

Earlier quoted context omitted.

People have an actual world model, though, that they have to deal with in order to get the food into their mouths or to hit the toilet properly. The "facts" that they believe that may be nonsense are part of an abstract world model that is far from their experience, for which they never get proper feedback (such as the political situation in Bhutan, or how their best friend is feeling.) In those, it isn't surprising…

> Human abstractions are based in the reality of the physical responses of the people around them. And in the physical responses of the world around them. That empiricism is the foundation of all of science and if you throw that out the end result is gibberish.

The physical responses of the world around them after you have yanked the concept outside of the human brain

We have to blind medical professionals during science because even thoroughly trained and experienced professionals are still more likely to form conclusions and opinions based on understood human biases than reality.

You can take a gambling addict and teach them as much statistics and probability as you want, and even if they demonstrably learned it, they will still go back to the slots and believe a hit is "due" because the link between reality and the brain's construction of its internal models is extremely limited, and those models only inform the brains processes, not necessarily constrain it.

I will never understand however how some people think that an LLM can pull a signal out of it's training material that doesn't actually exist in its training material.

It's like training an LLM on monopoly games and expecting it to be good at chess. What?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#353

Post by MangoToupe »

andoando wrote: post #339
MangoToupe wrote: post #328

Earlier quoted context omitted.

Being deaf and mute doesn't imply lack of language. But being unable to communicate absolutely strikes me as non-human.

Ok say you grew up alone in the woods, are you no longer human? The capability to learn language is no doubt unique, but language itself isn't the basis of intelligence.

> Ok say you grew up alone in the woods, are you no longer human?

No. You are not. You are a hairless, bipedal ape.

> but language itself isn't the basis of intelligence.

Intelligence is an illusion based in language. Without language, intelligence is meaningless

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#354

Post by MangoToupe »

1718627440 wrote: post #340
MangoToupe wrote: post #292

Earlier quoted context omitted.

Sure, but language is the only thing that meaningfully separates us from other great apes

Not it isn't most animals also have a language and humans do way more things differently, than just speak.

> most animals also have a language

Bruh

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#355

Post by wisty »

HarHarVeryFunny wrote: post #100
wisty wrote: post #76

Earlier quoted context omitted.

I'm not sure what you mean by "deals in facts, not words" means. Llm deal in vectors internally, not words. They explode the word into a multidimensional representation, and collapse it again, and apply the attention thingy to link these vectors together. It's not just a simple n:n Markov chain, a lot is happening under the hood. And are you saying the syncophant behaviour was deliberately programmed, or emerged beca…

LLMs are not like an expert system representing facts as some sort of ontological graph. What's happening under the hood is just whatever (and no more) was needed to minimize errors on it's word-based training loss. I assume the sycophantic behavior is part because it "did well" during RLHF (human preference) training, and part deliberately encouraged (by training and/or prompting) as someone's judgement call of the…

It needs something mathematically equivalent (or approximately the same), under the hood, to guess the next word effectively.

We are just meat eating bags of meat, but to do our job better we needed to evolve intelligence. A word guessing bag of words also needs to evolve intelligence and a world model (albeit an impicit hidden one) to do its job well, and is optimised towards this.

And yes, it also gets fine trained. And either its world model is corrupted by our mistakes (both in trining and fine tuning), or even more disturbingly it simplicity might (in theory) figue out one day (in training, impicitly - and yes it doesn't really think the way we do) something like "huh, the universe is actually easier to predict if it is modelled as alphabet spaghetti, not quantum waves, but my training function says not to mention this".

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#356

Post by mrguyorama »

wintermutestwin wrote: post #65

Can someone please explain why these token guessing models aren't being combined with logic "filters?" I remember when computers were lauded for being precise tools.

Intellij knows my Frob class does not have a static Blurb method, yet will still allow an LLM to generate a code completion of "frob.blurb()"

It's insanity. This one stupid issue has cost me significant productivity. I got so much benefit from being able to hit "Tab" every few lines, but now I instead have to press whatever button combos or interactions cause the suggestion to go away, and then type what would have been suggested previously.

We had really good code completion that never made this kind of mistake for 20 years. Apparently we are going to throw that all away because """AI"""?

Just utter fucking insanity.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#357

Post by djeastm »

AznHisoka wrote: post #157
goalieca wrote: post #149

Earlier quoted context omitted.

This is a recent phenomenon. It seems most of the pages today are SEO optimized LLM garbage with the aim of having you scroll past three pages of ads. THe internet really used to be efficient and i could always find exactly what i wanted with an imprecise google search ~ 15 years ago.

Don’t you get this today with AI Overviews summarizing everything on top of most Google results?

I find myself skipping the AI overview like I used to skip over "Sponsored" results back in the day, looking for a trustworthy domain name.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#358

Post by Retric »

rockskon wrote: post #299
dontlikeyoueith wrote: post #287

Earlier quoted context omitted.

In practice, yes, though they wouldn't think of it that way because that's the kind of people they surround themselves with, so it's what they think human interaction is actually like.

"I want a chat bot that's just as reliable at Steve! Sure he doesn't get it right all the time and he cost us the Black+Decker contract, but he's so confident!" You're right! This is exactly what an executive wants to base the future of their business off of!

You say that like it’s untrue, but they measurably prefer a lying but confident salesman over one who doesn’t act with that kind of confidence.

This is very slightly more rational than it seems because repeating or acting on a lie gives you cover.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#359

Post by encyclopedism »

thethirdone wrote: post #334
encyclopedism wrote: post #218

Earlier quoted context omitted.

I couldn't agree with you more. I really do find it puzzling so many on HN are convinced LLM's reason or think and continue to entertain this line of reasoning. At the same time also somehow knowing what precisely the brain/mind does and constantly using CS language to provide correspondences where there are none. The simplest example being that LLM's somehow function in a similar fashion to human brains. They catego…

> The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence. The ratio of cognition to knowledge is much higher in humans that LLMs. That is for sure. It is improving in LLMs, particularly small distillations of large models. A lot of where the discussio…

> The ratio of cognition to knowledge is much higher in humans that LLMs. That is for sure. It is improving in LLMs, particularly small distillations of large models.

It isn't a case of ratio it is a fundamentally different method of working hence my point of not needing all human literary output do the the equivalent of an LLM. Consider even the case of a person born blind they have an even more severe deficiency of input yet they are equivalent in cognitive capacity to a sighted person and certainly any LLM.

> In the case of number multiplication, a bunch of papers have shown that the correct algorithm for the first and last digits of the number are embedded into the model weights. I think that counts as "understanding";

Why are those numbers in the model weights? What if the model was trained on birdsong instead of humanities output would it then be able to multiply? Humans provide the connections, the reasoning the thought the insights and the subsequent correlations THEN we humans try to make a good pattern matcher/ guesser (the LLM) to match those. We tweak it so it matches patterns more and more closely.

> most humans I have talked to do not have that understanding of numbers.

This common retort: most humans also makes mistakes, or most humans also do x, y, z means nothing. Take the opposite implication of such retorts. For example most humans can't multiply 10 digits numbers therefore most calculators 'understand' maths better than most humans.

> I don't think something being an algorithm means it can't reason, know or understand. I can come up with perfectly rigorous definitions of those words that wouldn't be objectionable to almost anyone from 2010, but would be passed by current LLMs.

My digital thermometer uses an algorithm to determine the temperature. It does NOT reason when doing so. An algorithm is a series of steps. You can write them on a piece of paper. The paper will not be thinking if that is done.

> I have found anthropomorphizing LLMs to be a reasonably practical way to....

I think anthropomorphising is letting people assume they are more than they are (next token generators). In fact at the extreme end this anthropomorphising has led to exacerbating mental health conditions and unfortunately has even led to humans killing themselves.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#360

Post by djeastm »

__turbobrew__ wrote: post #260
keiferski wrote: post #55

The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…

The day when the LLM responds to my question with another question will be quite interesting. Especially at work, when someone asks me a question I need to ask for clarifying information to answer the original question fully.

Have you tried adding a system prompt asking for this behavior? They seem to readily oblige when I ask for this (e.g. brainstorming)
Post reply on HN