Live data from Hacker News

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

steerlabs.substack.com

381–390 of 399 posts

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#381
post #345

Earlier quoted context omitted.

"The LLM learns 2nd hand heresay, with no idea of what's true or false, what generalizations are valid, or what would be hallucinatory, " - do you know what is true and what is false? Take this: https://upload.wikimedia.org/wikipedia/commons/thumb/b/be/Ch... - Do you believe your eyes or do you believe the text about it?

I can experiment and verify, can't I ?

Do you? Do most? Do we for 99.999% of stuff we're taught?

Besides, the LLM can also "experiment and verify" some things now. E.g. it can spin up Python and run a script to verify some answers.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#382

Earlier quoted context omitted.

> LLMs are text model, not world models and that is the root cause of the problem. Is it though? In the end, the information in the training texts is a distilled proxy for the world, and the weighted model ends up being a world model, just an once-removed one. Text is not that different to visual information in that regard (and humans base their world model on both). > Not having a world model is a massive disadvanta…

>In the end, the information in the training texts is a distilled proxy for the world This is routinely asserted. How has it been proven ? Humans write all sorts of text that has zero connection to reality, even when they are ostensibly writing about reality . Training on ancient greek philosophy which was expressly written to distill knowledge about the real world would produce a stupid LLM that doesn't know about t…

I think a key insight from your comment is that in order to be able to verify whether the stuff we allow into our brains gets permanent billing we test it against our world model and if it does not fit we reject it. LLMs accept anything in the training set so curation of the training set is a big factor in the quality of the LLMs output. That's an incremental improvement, not a massive leap forward but it definitely will help to reduce the percentage of bullshit created.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#383

Earlier quoted context omitted.

I can experiment and verify, can't I ?

Do you? Do most? Do we for 99.999% of stuff we're taught? Besides, the LLM can also "experiment and verify" some things now. E.g. it can spin up Python and run a script to verify some answers.

I think if we're considering the nature of intelligence, pursuant to trying to replicate it, then the focus needs to be more evolutionary and functional, not the behavior of lazy modern humans who can get most of their survival needs met at Walmart or Amazon!

The way that animals (maybe think apes and dogs, etc, not just humans) learn is by observing and interacting. If something is new or behaves in unexpected ways then "prediction failure", aka surprise, leads to them focusing on it and interacting with it, which is the way evolution has discovered for them to learn more about it.

Yes, an LLM has some agency via tool use, and via tool output it can learn/verify to some extent, although without continual learning this is only of ephemeral value.

This is all a bit off topic to my original point though, which is the distinction between trying to learn from 2nd hand conflicting heresay (he said, she said) vs having the ability to learn the truth for yourself, which starts with being built to predict the truth (external real-world) rather than being built to predict statistical "he said, she said" continuations. Sure, you can mitigate a few of an LLM's shortcomings by giving them tools etc, but fundamentally they are just doing the wrong thing (self-prediction) if you are hoping for them to become AGI rather than just being language models.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#385
post #344

Earlier quoted context omitted.

> So it needs to know facts, albeit the currently accepted ones. Knowing the facts is a good way to compression data. It's not a compression engine - it's just a statistical predictor. Would it do better if it was incentivized to compress (i.e training loss rewarded compression as well as penalizing next-word errors)? I doubt it would make a lot of difference - presumably it'd end up throwing away the less frequently…

Both compression engines and llm work by assigning scores to the next token. If you can guess the probability distribution of the next token you have a near perfect text compressor, and a near perfect llm. Yeah in the real world they have different trade-offs. Here's a paper by deep mind. https://arxiv.org/pd7f/2309.10668 - titled LANGUAGE MODELING IS COMPRESSION

An LLM is a transformer of a specific size (number of layers, context width, etc), and ultimately number of parameters. A trillion parameter LLM is going to use all trillion parameters regardless of whether you train it on 100 samples or billions of them.

Neural nets, including transformers, learn by gradient descent, according to the error feedback (loss function) they are given. There is no magic happening. The only thing the neural net is optimizing for is minimizing errors on the loss function you give it. If the loss function is next-token error (as it is), then that is ALL it is optimizing for - you can philosophize about what they are doing under the hood, and write papers about that ("we advocate for viewing the prediction problem through the lens of compression"), but at the end of the day it is only pursuant to minimizing the loss. If you want to encourage compression, then you would need to give an incentive for that (change the loss function).

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#386
post #380

Earlier quoted context omitted.

> "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly point it out, and then all sorry-this and apologize-that and "here's the actual answer!" It's exactly the kind of personality you always see bubbling up into the orbit of a rich and powerful tech CEO. You ma…

Maybe it’s just the fact that many models are trained by americans? I’ve seen great improvements in answers by asking it to “tone it down, answer like you’re British”.

Oh interesting insight, I should try to see what that does. Jolly good old chap, let me check on why the laaabrary is on faaahre... ;)

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#388

Earlier quoted context omitted.

Yes, that is in fact their revealed preference. Did you have a point?

You use unfalsifiable logic. And you seem to argue that, given the choice, CEOs would prefer not to maximize revenue in favor of... what, affection for an imaginary intern?

Cute straw man.

You must be a CEO.

I'm not arguing anything. I'm observing reality. You're the one who is desperate to rationalize it.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#389

Earlier quoted context omitted.

You use unfalsifiable logic. And you seem to argue that, given the choice, CEOs would prefer not to maximize revenue in favor of... what, affection for an imaginary intern?

Cute straw man. You must be a CEO. I'm not arguing anything. I'm observing reality. You're the one who is desperate to rationalize it.

You are declaring your imagined logic as fact. Since I do not agree with the basis upon which you pin your argument on, there is no further point in discussion.

Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks

#390
post #158
post #133

I wish we didn't use LLMs to create test code. Tests should be the only thing written by a human. Let the AI handle the implementation so they pass!

Humans writing tests can only help against some subset of all problems that can happen with incompetent or misaligned LLMs. For example, they can game human-written and LLM-written tests just the same.

Not property-based tests. Either way, the human is there to tell the machine what to do: tests are one way of expressing that.
Post reply on HN