Does it matter that, like so much in the Math for AI sphere, core details seem to be totally bungled? e.g. see various comments in this thread https://x.com/waltonstevenj/status/1834327595862950207
LLMs Will Always Hallucinate, and We Need to Live with This
61–70 of 274 posts
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#62I'm of the opinion that the current architectures are fundamentally ridden with "hallucinations" that will severely limit their practical usage (including very much what the hype thinks they could do). But this article puts an impossible limit to what it is to "not-hallucinate". It essentially restates well known fundamental limitations of formal systems and mechanistic computation and then presents the trivial resul…
C.S. Peirce, who is known for characterizing abductive reasoning and had a considerable on John Sowa’s old school AI work, had an interesting take on this. I can’t fully do it justice, but essentially he held that both matter and mind are real, but aren’t dual. Rather, there is a smooth and continuous transition between the two. However, whatever the nature of mind and matter really is, we have convincing evidence of…
I don’t know the technical philosophy terms for this, but my simplistic way of thinking about it is that when I’m “seriously” talking (not just emitting thoughtless cliché phrases), I’m talking about something. And this is observable because sometimes I have an idea that I have trouble expressing in words, where I know that the words I’m saying are not properly expressing the idea that I have. (I mean — that’s happening right now!)
I don’t see how that could ever happen for an LLM, because all it does is express things in words, and all it knows is the words that people expressed things with. We know for sure, that’s just what the code does; there’s no question about the underlying mechanism, like there is with humans.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#63> By establishing the mathematical certainty of hallucinations, we challenge the prevailing notion that they can be fully mitigated Having a mathematical proof is nice, but honestly this whole misunderstanding could have been avoided if we'd just picked a different name for the concept of "producing false information in the course of generating probabilistic text". "Hallucination" makes it sound like something is goi…
Maybe with vanilla LLMs, but new LLM training paradigms include post-training with the explicit goal of avoiding over-confident answers to questions the LLM should not be confident about answering. So hallucination is a malfunction, just like any overconfident incorrect prediction by a model.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#64I'm of the opinion that the current architectures are fundamentally ridden with "hallucinations" that will severely limit their practical usage (including very much what the hype thinks they could do). But this article puts an impossible limit to what it is to "not-hallucinate". It essentially restates well known fundamental limitations of formal systems and mechanistic computation and then presents the trivial resul…
> fundamentally ridden with "hallucinations" that will severely limit their practical usage On the other hand, a LLM that got rid of "hallucinations" is basically just a thing that copy-paste at that point. The interesting properties from LLMs comes from the fact that it can kind of make things up but still make them believable.
> can kind of make things up but still make them believable
This is the definition of a bullshitter, by the way.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#65Disagree - https://arxiv.org/abs/2406.17642
We cover halting problem and intractable problems in the related work.
Of course LLMs cannot give answers to intractable problems.
I also don’t see why you should call an answer of “I cannot compute that” to a halting problem question a hallucination.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#66> By establishing the mathematical certainty of hallucinations, we challenge the prevailing notion that they can be fully mitigated Having a mathematical proof is nice, but honestly this whole misunderstanding could have been avoided if we'd just picked a different name for the concept of "producing false information in the course of generating probabilistic text". "Hallucination" makes it sound like something is goi…
Yes, exactly, it’s a post-facto value judgment, not a precise term. If I understand the meaning of the word, “hallucination” is all the model does . If it happens to hallucinate something we think is objectively true, we just decide not to call that a “hallucination”. But there’s literally no functional difference between that case and the case of the model saying something that’s objectively false, or something whos…
Exactly this, I've been saying this since the beginning. Every response is a hallucination - a probabilistic string of words divorced from any concept of truth or reality.
By total coincidence, some hallucinations happen to reflect the truth, but only because the training data happened to generally be truthful sentences. Therefore, creating something that imitates a truthful sentence will often happen to also be truthful, but there is absolutely no guarantee or any function that even attempts to enforce that.
All responses are hallucinations. Some hallucinations happen to overlap the truth.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#67LLM and other generative output can only be useful for a purpose or not useful. Creating a generative model that only produces absolute truths (as if this was possible, or there even were such a thing) would make them useless for creative pursuits, jokes, and many of the other purposes to which people want to put them. You can’t generate a cowboy frog emoji with a perfectly reality-faithful model.
To me this means two things:
1. Generative models can only be helpful for tasks where the user can already decide whether the output is useful. Retrieving a fact the user doesn’t already know is not one of those use cases. Making memes or emojis or stories that the user finds enjoyable might be. Writing pro forma texts that the user can proofread also might be.
2. There’s probably no successful business model for LLMs or generative models that is not already possible with the current generation of models. If you haven’t figured out a business model for an LLM that is “60% accurate” on some benchmark, there won’t be anything acceptable for an LLM that is “90% accurate”, so boiling yet another ocean to get there is not the golden path to profit. Rather, it will be up to companies and startups to create features that leverage the existing models and profit that way rather than investing in compute, etc.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#68Earlier quoted context omitted.
C.S. Peirce, who is known for characterizing abductive reasoning and had a considerable on John Sowa’s old school AI work, had an interesting take on this. I can’t fully do it justice, but essentially he held that both matter and mind are real, but aren’t dual. Rather, there is a smooth and continuous transition between the two. However, whatever the nature of mind and matter really is, we have convincing evidence of…
I don’t know the technical philosophy terms for this, but my simplistic way of thinking about it is that when I’m “seriously” talking (not just emitting thoughtless cliché phrases), I’m talking about something. And this is observable because sometimes I have an idea that I have trouble expressing in words, where I know that the words I’m saying are not properly expressing the idea that I have. (I mean — that’s happen…
> I have an idea that I have trouble expressing in words, where I know that the words I’m saying are not properly expressing the idea that I have. (I mean — that’s happening right now!)
That's a quality insight. Which, come to think of it, is an interestingly constructed word given what you just said.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#69Models are often wrong but sometimes useful. Models that provide answers couched in a certain level of confidence are miscalibrated when all answers are given confidently. New training paradigms attempt to better calibrate model confidence in post-training, but clearly there are competing incentives to give answers confidently given the economics of the AI arms race.
Re: LLMs Will Always Hallucinate, and We Need to Live with This
#70Isn’t hallucination just the result of speaking out loud the first possible answer to the question you’ve been asked? A human does not do this. First of all, most questions we have been asked before. We have made mistakes in answering them before, and we remember these, so we don’t repeat them. Secondly, we (at least some of us) think before we speak. We have an initial reaction to the question, and before expressing…
> So, to evaluate the intelligence of an LLM based on its first “gut reaction” to a prompt is probably misguided.
There's no intelligence to evaluate. They're not intelligent. There's no logic or cogitation in them.