It’s not really hallucinating though, is it? It’s repeating a pattern in its training data, which is wrong but is presented in that training data (and by the author of this piece, but unintentionally) as being the solution to the problem. So this has more in common with an attack than a hallucination on the LLM’s part.
It's more complicated than what happens with Markov chain models but you can use them to build an intuition for what's happening.
Imagine a very simple Markov model trained on these completely factual sentences:
- "The sky is blue and clear"
- "The ocean is blue and deep"
- "Roses are red and fragrant"
When the model is asked to generate text starting with "The roses are...", it might produce: "The roses are blue and deep"
This happens not because any training sentence contained incorrect information, but because the model learned statistical patterns from the text, as opposed to developing a world model based on physical environmental references.