Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

Post reply on HN
Print view View original

151–160 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#151

Post by jacquesm »

coldtea wrote: post #145
jacquesm wrote: post #140

LLMs are text model, not world models and that is the root cause of the problem. If you and I would be discussing furniture and for some reason you had assumed the furniture to be glued to the ceiling instead of standing on the floor (contrived example) then it would most likely only take one correction based on your actual experience that you are probably on the wrong track. An LLM will happily re-introduce that err…

> LLMs are text model, not world models and that is the root cause of the problem. Is it though? In the end, the information in the training texts is a distilled proxy for the world, and the weighted model ends up being a world model, just an once-removed one. Text is not that different to visual information in that regard (and humans base their world model on both). > Not having a world model is a massive disadvanta…

I really think it is, this is the exact same thing that keeps going wrong in these conversations over-and-over again. There simply is no common sense, none at all, just a likelihood of applicability. To the point that I even wonder how it is possible to get such basic stuff for which there is an insane amount of support wrong.

I've had an hour long session which essentially revolved around why the landing gear of an aircraft is at the bottom, not at the top of the vehicle (paraphrased for good reasons but it was really that basic). And this happened not just once, but multiple times. Confident declarations followed by absolute nonsense, I've even had - I think it was ChatGPT - try to gaslight me with something along the lines of 'you yourself said' on something that I did not say (this is probably the most person like thing I've seen it do).

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#152

Post by lukev »

nickdothutton wrote: post #44

- Claude, please optimise the project for performance. o Claude goes away for 15 minutes, doesn't profile anything, many code changes. o Announces project now performs much better, saving 70% CPU. - Claude, test the performance. o Performance is 1% _slower_ than previous. - Claude, can I have a refund for the $15 you just wasted? o [Claude waffles], "no".

If you provide it a benchmark script (or ask it to write one) so it has concrete numbers to go off of, it will do a better job.

I'm not saying these things don't hallucinate constantly, they do. But you can steer them toward better output by giving them better input.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#153

Post by anal_reactor »

throw4847285 wrote: post #94
anal_reactor wrote: post #83

Earlier quoted context omitted.

This is exactly why I don't like dealing with most people.

Every thread like this I like to go through and count how many people are making the pro-AI "Argument from Misanthropy." Based on this exercise, I believe that the biggest AI boosters are simply the most disagreeable people in the industry, temperamentally speaking.

Just because I'm disagreeable it doesn't mean I'm wrong.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#155

Post by coldtea »

HarHarVeryFunny wrote: post #24
jqpabc123 wrote: post #2

We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.

The factuality problem with LLMs isn't because they are non-deterministic or statistically based, but simply because they operate at the level of words, not facts. They are language models. You can't blame an LLM for getting the facts wrong, or hallucinating, when by design they don't even attempt to store facts in the first place. All they store are language statistics, boiling down to "with preceding context X, mos…

>but simply because they operate at the level of words, not facts. They are language models.

Facts can be encoded as words. That's something we also do a lot for facts we learn, gather, and convey to other people. 99% of university is learning facts and theories and concept from reading and listening to words.

Also, even when directly observing the same fact, it can be interpreted by different people in different ways, whether this happens as raw "thought" or at the conscious verbal level. And that's before we even add value judgements to it.

>All they store are language statistics, boiling down to "with preceding context X, most statistically likely next words are A, B or C".

And how do we know we don't do something very similar with our facts - make a map of facts and concepts and weights between them for retrieving them and associating them? Even encoding in a similar way what we think of as our "analytic understanding".

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#156

Post by nosianu »

MangoToupe wrote: post #109
motoboi wrote: post #98

Earlier quoted context omitted.

Reflect a moment over the fact that LLMs currently are just text generators. Also that the conversational behavior we see it’s just examples of conversations that we have the model to mimic so when we say “System: you are a helpful assistant. User: let’s talk. Assistant:” it will complete the text in a way that mimics a conversation?. Yeah, we improved over that using reinforcement learning to steer the text generati…

> Reflect a moment over the fact that LLMs currently are just text generators. You could say the same thing about humans.

The human world model is based on physical sensors and actions. LLMs are based on our formal text communication. Very different!

Just yesterday I observed myself acting on an external stimulus without any internal words (this happens continuously, but it is hard to notice because we usually don't pay attention to how we do things): I sat in a waiting area of a cinema. A woman walked by and dropped her scarf without noticing. I automatically without thinking raised arm and pointer finger towards her, and when I had her attention pointed behind her. I did not have time to think even a single word while that happened.

Most of what we do does not involved any words or even just "symbols", not even internally. Instead, it is a neural signal from sensors into the brain, doing some loops, directly to muscle activation. Without going through the add-on complexity of language, or even "symbols".

Our word generator is not the core of our being, it is an add-on. When we generate words it's also very far from being a direct representation of internal state. Instead, we have to meander and iterate to come up with appropriate words for an internal state we are not even quite aware of. That's why artists came up with all kinds of experiments to better represent our internal state, because people always knew the words we produce don't represent it very well.

That is also how people always get into arguments about definitions. Because the words are secondary, and the further from the center of established meaning for some word you get the more the differences show between various people. (The best option is to drop insisting of words being the center of the universe, even just the human universe, and/or to choose words that have the subject of discussion more firmly in the center of their established use).

We are text generators in some areas, I don't doubt that. Just a few months ago I listened to some guy speaking to a small rally. I am certain that not a single sentence he said was of his own making, he was just using things he had read and parroted them (as a former East German, I know enough Marx/Engels/Lenin to recognize it). I don't want to single that person out, we all have those moments, when we speak about things we don't have any experiences with. We read text, and when prompted we regurgitate a version of it. In those moments we are probably closest to LLM output. When prompted, we cannot fall back on generating fresh text from our own actual experience, instead we keep using text we heard or read, with only very superficial understanding, and as soon as an actual expert shows up we become defensive and try to change the context frame.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#157

Post by AznHisoka »

goalieca wrote: post #149
jodrellblank wrote: post #103

Earlier quoted context omitted.

> LMs don't do this. Instead, every question is immediately responded with extreme confidence with a paragraph or more of text. Having just read a load of Quora answers like this, which did not cover the thing I was looking for, that is how humans on the internet behave and how people have to write books, blog posts, articles, documentation. Without the "dance" to choose a path through a topic on the fly, the author…

This is a recent phenomenon. It seems most of the pages today are SEO optimized LLM garbage with the aim of having you scroll past three pages of ads. THe internet really used to be efficient and i could always find exactly what i wanted with an imprecise google search ~ 15 years ago.

Don’t you get this today with AI Overviews summarizing everything on top of most Google results?

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#158

Post by lxgr »

tasuki wrote: post #133

I wish we didn't use LLMs to create test code. Tests should be the only thing written by a human. Let the AI handle the implementation so they pass!

Humans writing tests can only help against some subset of all problems that can happen with incompetent or misaligned LLMs. For example, they can game human-written and LLM-written tests just the same.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#160

Post by coldtea »

HarHarVeryFunny wrote: post #89
AlecSchueler wrote: post #64

Earlier quoted context omitted.

In a way though those things aren't so different as they might first appear. The factual answer is traditionally the most plausible response to many questions. They don't operate on any level other than pure language but there are a heap of behaviours which emerge from that.

> The factual answer is traditionally the most plausible response to many questions Except in cases where the training data is more wrong than correct (e.g. niche expertise where the vox pop is wrong). However, an LLM no more deals in Q&A than in facts. It only typically replies to a question with an answer because that itself is statistically most likely, and the words of the answer are just selected one at a time i…

>Except in cases where the training data is more wrong than correct (e.g. niche expertise where the vox pop is wrong)

Same for human knowledge though. Learn from society/school/etc that X is Y, and you repeat X is Y, even if it's not.

>However, an LLM no more deals in Q&A than in facts. It only typically replies to a question with an answer because that itself is statistically most likely, and the words of the answer are just selected one at a time in normal LLM fashion.

And how is that different than how we build up an answer? Do we have a "correct facts" repository with fixed answers to every possibly question, or we just assemble our training data from a weighted graph (or holographic) store of factoids and memories, and our answers are also non deterministic?

Post reply on HN