111–120 of 491 posts
Ah, the AI hype is now entering the "let's be real" phase. Haven't seen a frenzied post on alignment in a while now.
I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…
When we say “know” it usually means being factual. For an LLM to ‘know’ it doesn’t know, it would have had to move away from pure correlations on words, and meta processing about its own results.
I can see this happen with two LLMs working together (and there are Evals that use just this), however each LLM still has no self awareness of its limits.
This was a terribly convoluted argument to make.
It seems a stretch to call it "inevitable". "Inevitable given the current architecture without modifications" at most. Also, I'm missing a section on how (if) human brains manage to avoid hallucinations in this. Also, it doesn't have to never hallucinate, it just has to hallucinate less than we do.
According to Buddhist philosophy, our whole identity is a hallucination :) I kind of concur.
Earlier quoted context omitted.
The model will just hallucinate scores, they don't have the ability to self reflect with words like that, there is no function for it to associate the word 90 with its own mental score 90% so anything it will say about those scores is just a hallucination.
Try this out: After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `certainty', all in range 0 to 1, where 0 is the worst, and 1 is the best. Strive for highest score possible. Make sure the rating is the last thing written as to be parsed by machine. The question is: make and explain 20-year predictions of the geopolitical fut…
In general, I think most people are not aware they are extending reasoning tools they use with human generated content, to LLM generated content.
This leads to assumptions on things like “the LLM knows”, “the LLM understands”, etc.
Earlier quoted context omitted.
LLMs don't have a concept of sources for their statements. Ask them to give you some literature recommendations on something it has explained to you. You'll get plenty of plausible sounding papers that don't exist. Humans know to some extent why they know (read it in a text book, colleague mentioned it). LLMs don't seem to.
Ask a human to provide accurate citations for any random thing they know and they won't be able to do a good job either. They'd probably have to search to find it, even if they know they got it from a document originally and have some clear memory of what it said.
This argument comes up many times “people don’t do this” - but that is a question of frequency, not whether or not people are capable.
Earlier quoted context omitted.
That is correct, it's always hallucinating and making things up. Just because those hallucinations sometimes randomly happens to be right, people concluded that being wrong is the exception, while being right is somehow the rule. It's like when people read [insert millenias old text here], finds a part that happens to illustrate something in their life today and conclude that it is a prophecy that predicted the futur…
For heavy LLM users, there is probably a dopamine hit when it does something right, much as gamblers get a hit when the fruit machine pays out. Perhaps LLM use is no more productive than gambling, and perhaps can be abused in a similar way.
Earlier quoted context omitted.
Asking it to write code for you is basically asking it to hallucinate.
I don't think so. I think it's asking it to repeat code it has been trained on
LLMs are just generating tokens. Hallucination perpetuates an unhelpful anthropomorphization of LLMs.
For production systems the considerations are: - how often does it error? - how bad are the errors? - how tolerable are the errors? - how detectable are the errors? - how recoverable are the errors? For example, a pocket calculator that occasionally was out at the 3rd decimal place might do much more damage than one that quite regularly returned NaN. LLMs error both regularly and badly, so detectability and recoverab…
At that point, you know its not thinking, its doing token prediction.
Earlier quoted context omitted.
The model will just hallucinate scores, they don't have the ability to self reflect with words like that, there is no function for it to associate the word 90 with its own mental score 90% so anything it will say about those scores is just a hallucination.
Try this out: After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `certainty', all in range 0 to 1, where 0 is the worst, and 1 is the best. Strive for highest score possible. Make sure the rating is the last thing written as to be parsed by machine. The question is: make and explain 20-year predictions of the geopolitical fut…