Earlier quoted context omitted.
If an LLM happens to know the answer to your question, that answer will have the greatest weight, and will therefore become a non-hallucinated output. Otherwise the output will be hallucinated. Note that a hallucination may manifest as an attempt to extrapolate, which may be successful. If you query an LLM with prior knowledge that the LLM doesn’t know the answer, you are guaranteed to receive a hallucinated output.…
But that's not how they actually work. > "If an LLM happens to know the answer to your question, that answer will have the greatest weight " An LLM doesn’t “know” anything in the way you’re imagining. It doesn’t have stored facts or indexed knowledge to check against, it just has weights learned between token sequences, and it outputs whatever next token is assigned the highest probability given the prompt and prior…
Neither does your brain and yet you do "know" something.
> but it might just as easily produce a plausible-sounding but wrong one, even if the correct information was in the training data
If the majority of information that was in the LLM's training data said 1 + 1 = 3, the LLM will tell you that 1 + 1 = 3, even if there was some information that said 1 + 1 = 2, and there's nothing wrong with that because the LLM is not supposed to fact-check.
> the model’s training biases it toward correct sequences more often than incorrect ones
No, the model's training biases it toward sequences that appear more frequently.