141–150 of 318 posts
Why is everyone expecting LLMs to be like the Star Trek computer? I wonder if anyone's ever measured what the hallucination rate of a human is.
Because AI company executives and devoted vibecoders constantly make egregious claims like "programming is fully solved" and even straight up "hallucinations don't exist on frontier models"
Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to…
Earlier quoted context omitted.
Because AI company executives and devoted vibecoders constantly make egregious claims like "programming is fully solved" and even straight up "hallucinations don't exist on frontier models"
We don't have to listen to these people and can form our own perspectives. Following bad leaders is something to avoid
Earlier quoted context omitted.
There is no concept of "knowledge" in LLM as it is on Wikipedia. The question-tokens define the answer-tokens. That's it. The art relies in clustering the relevant weights together.
> There is no concept of "knowledge" in LLM as it is on Wikipedia. There can be, you don't know if the closed source models aren't using something like DeepSeek's Engram.
While DeepSeek describe this as "knowledge lookup", what Engram is really trying to do is separate dynamic reasoning from static pattern recall, with the static patterns just being word-level n-gram statistics, not declarative facts/knowledge.
Just because 2-3 words often appear together in a sequence doesn't mean they represent a fact or truth (or falsehood) - it is just an n-gram statistical regularity.
If Engram helps reduce LLM GPU memory and FLOP requirements then that is great, but it's not a solution for Hallucination.
Why is everyone expecting LLMs to be like the Star Trek computer? I wonder if anyone's ever measured what the hallucination rate of a human is.
Earlier quoted context omitted.
I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…
Sucky human-written code is still based on human understanding, which can change over time, be readjusted or solidified. People implement something wrong once, then update their perspective, then in the future does it right. LLMs doesn't have this benefit. You forget to add the correct to the system prompt, and the LLM will repeat the same mistake over and over, and worse than that, their mistakes aren't based on the…
Whilst I don't claim any true "understanding" as that is a very loaded term that doesn't mean it's just random guesses.
Anyone using recent LLM coding agents on a regular basis would probably agree that there's something going on that fits some non-athropomorphizing, non-sentience-assigning definition of "understanding"
As for the point about improvement - I think that's an orthogonal issue to the overall code quality. With regard to human codebases - there's plenty of scenarios that negate the improvement of individuals. We're comparing organizations with LLMs - not individuals with LLMs and that makes a significant difference.
One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…
If we had a theoretical technique to identify the true and objective reality we'd use it in the courts and laboritories. There is no such technique, but what we do have is 2 techniques that seem work: 1) Has a certain standard of evidence been met? 2) Are the related arguments free of logical inconsistencies? We can train the LLMs to do 2, and maybe even 1 to some extent (exactly what quality of evidence a computer c…
But even in muddy fields of reality like medicine, there are objective facts to be found. When someone comes into an ER with chest pain, you often find a true, undeniable reason for why that is happening. If their lung has collapsed, a coronary artery is clogged or the aortic artery is dissecting, even if you don't find that out it tends to be clear in retrospect. The area of reality that becomes muddy is when use proxy signals to try to figure out who gets promoted to expensive/harmful examinations we can make final conclusions from, or the cases that don't fit cleanly into one bucket or the other. But very often, the gold standard truly is golden.
Of course, many realms of reality cannot be verified in this way. But I'd argue that there are quite a few that can.
Calling llm slop "hallucinating" is so counter-productive imo. After all, LLMs are just a variant of markov chains and as such this technology isn't able to discern falsehoods from truths. It's like trying to use a barometer to tell the time.
You are also just a variant of markov chains wired in your brain. So what you complaining about?
> it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may stil…
One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…
Training an extra "don't know" token means you have to build a moat between every other token. Between "yes" and "no", you don't have a muddled noisy area where both "yes" and "no" have relatively high probabilities, you need a new peak where "don't know" is higher. Then you just have new muddled areas between "yes" and "don't know", and "don't know" and "no". That requires even more finesse to train another answer in between.
Instead, you could check whether multiple options are about equally likely. But then you have to check if they are actually synonyms, like are the top two choices "Genève" and "Geneva", which is a good sign that the model knows the answer? Or are the top two "yes" and "no"?