>We are trying to fix probability with more probability. That is a losing game.
>We need to re-introduce Determinism into the stack.
>If it fails lets inject more prompts but call it "rules" and run the magic box again
Bravo.
The "confident idiot" problem: Why AI needs hard rules, not vibe checks
281–290 of 399 posts
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#282The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…
> LMs don't do this. Instead, every question is immediately responded with extreme confidence with a paragraph or more of text. Having just read a load of Quora answers like this, which did not cover the thing I was looking for, that is how humans on the internet behave and how people have to write books, blog posts, articles, documentation. Without the "dance" to choose a path through a topic on the fly, the author…
That's part of it, but I think another part is just the way the LLMs are tuned. They're capable of more conversational tones, but human feedback in post-training biases them toward a writing style that's more of a Quora / StackOverflow / Reddit Q&A style because that's what gets the best ratings during the RLHF process.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#283The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…
By default they don't ask questions. You can craft that behaviour with the system message or account settings. Though they will tend to ask 20 questions at once so you have to request it to limit to one question at a time to get a more natural experience.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#284I had been working on NLP, NLU mostly, some years before LLMs. I've tried the universal sentence encoder alongside many ML "techniques" in order to understand user intentions and extract entities from text. The first time I tried chatgpt that was the thing that surprised me most, the way it understood my queries. I think that the spotlight is on the "generative" side of this technology and we're not giving the query…
Yes, I was (and still am) similarly impressed with LLMs ability to understand the intent of my queries and requests.
I've tried several times to understand the "multi-head attention" mechanism that powers this understanding, but I'm yet to build a deep intuition.
Is there any research or expository papers that talk about this "understanding" aspect specifically? How could we measure understand without generation? Are there benchmarks out there specifically designed to test deep/nuanced understanding skills?
Any pointers or recommended reading would be much appreciated.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#285Earlier quoted context omitted.
It means you are not representative of humanity as a whole. You are likely in a small minority of people on an extreme of the personality spectrum. Any attempts to glibly dismiss critiques of AI with a phrase equivalent to "well I hate people" should be glibly dismissed in turn.
Maybe let's try to rectify the discussion. I think that current generation of LLMs displays astounding similarity to human behaviour. I'm not trying to dismiss issues with LLMs, I'm trying to point out the practicality of treating LLMs as awkward humans rather than programs. Yes, I hate people. But usually whenever there's a critique of LLMs, I can find a parallel issue in people. The extension is that "if people can…
The problem is one of negative polarization. I found myself skeptical of a lot of the claims around LLMs, but was annoyed by AI critics forming an angry mob anytime AI was used for anything. However, I still considered myself in that camp, and ended up far more annoyed by AI boosterism than AI skepticism, which pushed me in the direction of being even more negative about AI than I started. It's the mirror of what happened to you, as far as I can tell. And I'm sure both are very common, though admitting it makes one seem reactive rather than rational and so we don't talk about it.
However, I do dispute your central claim that the issues with LLMs parallel the issues with people. I think that's a very dehumanizing and self-defeating perspective. The only ethical system that is rational is one in which humans have more than instrumental value to each other.
So when critics divide LLMs and humans, sure, there is a descriptive element of trying to be precise about what human thought is, and how it is different than LLMs, etc. But there is also a prescriptive argument that people are embarrassed to make, which is that human beings have to be afforded a certain kind of dignity and there is no reason to extend that to an LLM based on everything we understand about how they function. So if a person screws up your order at a restaurant, or your coworker makes a mistake when coding, you should treat them with charitability and empathy.
I'm sure this sounds silly to you, but it shouldn't. The bedrock of the Enlightenment project was that scientific inquiry would lead to human flourishing. That's humanism. If we've somehow strayed so far from that, such that appeals to human dignity don't make sense anymore, I don't know what to say.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#286Earlier quoted context omitted.
You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.
Do you honestly think an executive wanted a chat bot that confidently lies?
No, but they like the sycophancy.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#287Earlier quoted context omitted.
You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.
Do you honestly think an executive wanted a chat bot that confidently lies?
In practice, yes, though they wouldn't think of it that way because that's the kind of people they surround themselves with, so it's what they think human interaction is actually like.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#288The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…
LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly…
> "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly point it out, and then all sorry-this and apologize-that and "here's the actual answer!" It's exactly the kind of personality you always see bubbling up into the orbit of a rich and powerful tech CEO.
You may be on to something there: the guys and gals that build this stuff may very well be imbibing these products with the kind of attitude that they like to see in their subordinates. They're cosplaying the 'eager to please' element to the point of massive irritation and left out the one feature that could serve to redeem such behavior which is competence.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#289Earlier quoted context omitted.
You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.
Do you honestly think an executive wanted a chat bot that confidently lies?
Given the matrix 'competent/incompetent' / 'sycophant/critic' I would not take it as read that the 'incompetent/sycophant' quadrant would have no adherents, and I would not be surprised if it was the dominant one.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#290Earlier quoted context omitted.
Nah. Talking like an LLM would get you fired in a day. People are already suspicious of ass-kissers, they hate it when they think people are not listening to them, and if you're an ass-kisser who's not listening and is then wrong about everything, they want you escorted out by security. The real human position would be to be an ass-kisser who hangs on every word you say, asks flattering questions to keep you talking,…
[flagged]
[flagged]