I find it interesting that we say "hallucination" when an LLM is wrong and, when we are wrong we simply made a mistake.
Yes, although when I'm wrong I rarely say "Oh, now I see. This is definitely it!"
LLM Hallucinations in Practical Code Generation
21–30 of 34 posts
Re: LLM Hallucinations in Practical Code Generation
#22I arrived at a similar conclusion through trial and error. Giving one-shot examples of code as context has given me the best results.
Re: LLM Hallucinations in Practical Code Generation
#23I still don't think hallucinations in generated code matter very much. They show up the moment you try to run the code, and with the current batch of "coding agent" systems it's the LLM itself that spots the error when it attempts to run the code. I was surprised that this paper talked more about RAG solutions than tool-use based solutions. Those seem to me like a proven solution at this point.
I'm surprised to read that from a prominent figure in the industry such as yourself. The problem is that many hallucinations do not produce a runtime error, and can be very difficult to spot by a human, even if the code is thoroughly reviewed, which in many cases doesn't happen. These can introduce security issues, do completely different things from what the user asked (or didn't ask), do things inefficiently, ignor…
If you define "hallucinations" to mean "any mistakes at all" then yes, a compiler won't catch them for you.
I define hallucinations as a a particular class of mistakes where the LLM invents eg a function or method that does not exist. Those are solved by ensuring the code runs. I wrote more about that here: https://simonwillison.net/2025/Mar/2/hallucinations-in-code/
Even beyond that more narrow definition of a hallucination, tool use is relevant to general mistakes made by an LLM. The new Phoenix.new coding agent actively tests the web applications it is writing using a headless browser, for example: https://simonwillison.net/2025/Jun/23/phoenix-new/
The more tools like this come into play, the less concern I have about the big black box of matrices occasionally hallucinating up some code that is broken in obvious or subtle ways.
It's still on us as the end users to confirm that the code written for us actually does the job we set out to solve. I'm fine with that too.
Re: LLM Hallucinations in Practical Code Generation
#24I still don't think hallucinations in generated code matter very much. They show up the moment you try to run the code, and with the current batch of "coding agent" systems it's the LLM itself that spots the error when it attempts to run the code. I was surprised that this paper talked more about RAG solutions than tool-use based solutions. Those seem to me like a proven solution at this point.
Interesting. How do existing systems catch Task Requirement hallucinations?
They don't. My comment was about "hallucinations in generated code".
Re: LLM Hallucinations in Practical Code Generation
#25I still don't think hallucinations in generated code matter very much. They show up the moment you try to run the code, and with the current batch of "coding agent" systems it's the LLM itself that spots the error when it attempts to run the code. I was surprised that this paper talked more about RAG solutions than tool-use based solutions. Those seem to me like a proven solution at this point.
"I still don't think hallucinations in generated code matter very much"
Tell these our Python developers who don't test anything outside of a narrow happy path.
Re: LLM Hallucinations in Practical Code Generation
#26I still don't think hallucinations in generated code matter very much. They show up the moment you try to run the code, and with the current batch of "coding agent" systems it's the LLM itself that spots the error when it attempts to run the code. I was surprised that this paper talked more about RAG solutions than tool-use based solutions. Those seem to me like a proven solution at this point.
"I still don't think hallucinations in generated code matter very much" Tell these our Python developers who don't test anything outside of a narrow happy path.
Yeah, if you use LLMs to write code and don't test outside the happy path you're vibe coding, you're not software engineering.
Re: LLM Hallucinations in Practical Code Generation
#27Re: LLM Hallucinations in Practical Code Generation
#28Earlier quoted context omitted.
I'm surprised to read that from a prominent figure in the industry such as yourself. The problem is that many hallucinations do not produce a runtime error, and can be very difficult to spot by a human, even if the code is thoroughly reviewed, which in many cases doesn't happen. These can introduce security issues, do completely different things from what the user asked (or didn't ask), do things inefficiently, ignor…
If you define "hallucinations" to mean "any mistakes at all" then yes, a compiler won't catch them for you. I define hallucinations as a a particular class of mistakes where the LLM invents eg a function or method that does not exist. Those are solved by ensuring the code runs. I wrote more about that here: https://simonwillison.net/2025/Mar/2/hallucinations-in-code/ Even beyond that more narrow definition of a hallu…
I think the more general/useful definition of "hallucination" is anytime the LLM predicts next word based on "least worst" (statistically) choice rather than based on any closely matching samples in the training data.
The LLM has to generate some word each time it is called, and unless it recognizes soon enough that "I don't know" is the best answer (in of itself problematic, since any such prediction would be based on the training data, not the LLM's own aggregate knowledge!), then it may back itself into a corner where it has no well-grounded continuation, but nonetheless has to spit out the statistically best prediction, even if that is a very bad ungrounded prediction such as a non-existent API, "fits the profile" concocted answer, or anything else ...
Of course the LLM's output builds on itself, so any ungrounded/hallucinated output doesn't need to be limited to a single word or API call, but may instead consist of a whole "just trying my best" sentence or chunk of code (better hope you have unit test code coverage to test/catch it).
Re: LLM Hallucinations in Practical Code Generation
#29Earlier quoted context omitted.
> The problem is that many hallucinations do not produce a runtime error [...] Don't hallucinations mean nonexistent things, that is, in the case of code: functions, classes, etc. How could they fail to lead to a runtime error, then? The fact that LLMs can produce unreliable or inefficient code is a different problem, isn't it?
Hallucinations can be manifested in different ways. Using nonexistent APIs is just one of them. The LLM could just as well hallucinate code that doesn't fix a problem, or hallucinate that a problem exists in the first place, all while using existing APIs. This might not be a major issue for tasks like programming where humans can relatively easily verify the output, but in other scientific fields this can be much mor…
You can test every line of code in your program, but how many people actually do?
It's one thing if you are just creating a throwaway prototype, or something so simple that you will naturally exercise 100% of the code when testing it, but when you start building anything non-trivial it's easy to have many code paths/flows that are rarely executed or tested. Maybe you wrote unit tests for all the obvious corner cases, but did you consider the code correctness when conditions A, then B, then C ... occurs?). Even 100% code coverage (every line of code tested) isn't going to help you there.
Re: LLM Hallucinations in Practical Code Generation
#30Earlier quoted context omitted.
Hallucinations can be manifested in different ways. Using nonexistent APIs is just one of them. The LLM could just as well hallucinate code that doesn't fix a problem, or hallucinate that a problem exists in the first place, all while using existing APIs. This might not be a major issue for tasks like programming where humans can relatively easily verify the output, but in other scientific fields this can be much mor…
You can test every line of code in your program, but how many people actually do? It's one thing if you are just creating a throwaway prototype, or something so simple that you will naturally exercise 100% of the code when testing it, but when you start building anything non-trivial it's easy to have many code paths/flows that are rarely executed or tested. Maybe you wrote unit tests for all the obvious corner cases,…
> You can test every line of code in your program, but how many people actually do?
In my mind, that's what separates genuinely excellent professional programmers from everybody else.