111–120 of 318 posts
Calling llm slop "hallucinating" is so counter-productive imo. After all, LLMs are just a variant of markov chains and as such this technology isn't able to discern falsehoods from truths. It's like trying to use a barometer to tell the time.
> It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer. DeepSeek V4 Pro (1.6T params, 49B active, 44 AA Intelligence Index score) has a ludicrous 94% hallucination score on the AA-Omniscience benchmark, meaning on questions that it couldn’t figure out, it only stated that it didn’t know around 6% of the time, and the rest it conf…
There is no concept of "knowledge" in LLM as it is on Wikipedia. The question-tokens define the answer-tokens. That's it. The art relies in clustering the relevant weights together.
There can be, you don't know if the closed source models aren't using something like DeepSeek's Engram.
> Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse
These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may still hallucinate more but they all hallucinate much less than the original 175B ChatGPT which was smaller and trained on (much) less data than anything current.
Edit: My mention of data comes from this quote:
> A shift is happening among major AI labs, who are becoming increasingly skeptical of endless parameter count and training data scaling
My take on the current situation: it seems clear that the industry has seen that there is still a lot left to squeeze out of sub-1T models. But for that you do need more, high-quality data in the distribution which you want to unlock capabilities for.
One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…
But if an LLM says "I don't know" should you pay for the tokens?
Earlier quoted context omitted.
Since models just output the the most probable tokens and you can never accuse them of doing anything other than making it all up, I would like to see these tests run with a prompt that attempts to mitigate hallucination and finishes with something like: "Telling me that you don't have the relevant information or that the task is impossible is extremely useful to me and a valid answer", and see how much that changes…
> In other words, you shouldn't choose the model that hallucinates the least without detailed prompting You're prompting it wrong is quickly becoming the new, you're holding it wrong. It's wild how willing software engineers are to blame the user when the actual problem is their own defective design. Ideally we all, as an industry, will stop accepting this as reasonable excuse for the demonstrated incompetence
Now granted, if the boat salesmen were pushing hard on the idea that the boat would fly and even put little wings on the side and I bought the boat I might get really angry when I found out that it didn't fly. And I might angrily storm into the salesroom yelling about how the design is defective. But if someone pointed out 'hey, it's a boat perhaps you should stick to sailing around in it and stop getting your undies in a bundle about it not flying' the correct response is probably to take a closer look, ignore the salesmen, and cruise around the lake. LLM's are quite handy at some things and have some weird limits. Learn the limits, enjoy your time at sea.
> it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may stil…
Earlier quoted context omitted.
It should be 1 for correct, 0 for don't know and -1 for wrong. They are much better incentives. In real life a wrong answer is much more damaging than a don't know.
See, this, to me, seems obvious, but I’m sure it’s more challenging/complex than I can imagine (I am NOT an expert on AI in any way imaginable). But there has to be a solution. Just yesterday I was asking Gemini to tell me about a certain college professor, and it gave me a list of facts about them. And it was perfect. Then, out of curiosity, I followed up with “tell me more about him!” and it spit out several more b…
If your example had "Validate any details before sharing them with the user, with multiple sources" as the system prompt, it was using a model that is strong at following system prompts precisely and had access to some basic tools, then it'd spend maybe minutes more, but the answer would have been way more accurate.
But no, Google want "the new search results" (LLM hallucinations) to be on top, so we end up with "sounds plausible" answers instead "Collection of evidence from reliable/semi-reliable" or similar, which sucks. We could have quality, but it's too expensive/slow, so we get slop instead, just to maximize for speed and convenience.
Earlier quoted context omitted.
> We can penalise models for incorrect answers on a wide range of questions, but this doesn’t lead to the emergence of a coherent worldview, which, coupled with logical abilities, is the only true remedy against hallucinations I don't think anyone is trying to add "a coherent worldview" by reducing hallucinations, not sure how that even realistically could be aim. What people want, is for the models to stop giving co…
Model outputs don't have a confidence score.
Earlier quoted context omitted.
Because nearly all benchmarks measure "accuracy" by giving you a point for a correct answer, and 0 points for everything else. If you have 100 questions you are 10% certain on, answering "I don't know" to all of those leads to 0 points, answering all of them as if you are confident leads to an expected value of 10 points. So that's what most AIs are trained to do AA-Omniscience is the only AI benchmark I know of wher…
It should be 1 for correct, 0 for don't know and -1 for wrong. They are much better incentives. In real life a wrong answer is much more damaging than a don't know.
Earlier quoted context omitted.
Depends on what your understanding of the product is. If someone sold you a "Solved all your problems" machine, and it suddenly doesn't solve all your problems, then probably no, you shouldn't pay. But the way I'm being sold LLMs, is basically "A text generator that gives your plausible-sounding human text that sometimes hallucinates and gets things wrong, based on your input", then regardless of what the outcome is,…
Let's pretend I am someone who has heard people talk about ChatGPT, but have on idea what it actually is. I go to the website and am not presented with any information, just a prompt. So I ask it what it is and what it can do for me. My ask: > In a couple sentences, explain to me the product I'm being sold with ChatGPT. What does it do for me? The Reply from ChatGPT: > ChatGPT is a conversational AI that helps you th…
> Can I trust the output you give me?
And I assume it explains what to trust VS not.
I think in the bottom you should also see something like "Any text can contain mistakes" or similar too, which I know is a far cry from what some people push in the press in regards to capabilities, but I still don't see the platforms themselves as lying about this, while I do see a bunch of people constantly over-hyping the possibilities.