Because hallucinations is something that from a distance looks very unimportant, but when looked closely is a structural problem. Some people here live very close to the LLM field.
Structural because while a human being can be the judge of an LLM output, a computer (or another LLM) cannot.
No amount of error correction is enough to turn an LLM output into a reliable input to another (possible dumb) computer system. Worse: each time that output is processed the error increases and when the final output is shown to an user, the error might have been amplified beyond human recovery (or recognition) capacity.
Think about this: one user sends Amazon support an email asking to refund for a stolen item.
Can this email be processed do feed an automatic refund pipeline system? If the answer is no and you need a human to verify the result, then we have one reason why hallucinations matter.
And there are the cases where a user verification is not even possible, like:
- what is the procedure to perform CPR in a person above 80 years old?
The user can’t recover errors in the output generated by an LLM here, because she doesn’t know the correct answer.
That being the case, you cannot build a search engine out of an LLM here. Hence hallucinations matters very much is this case too.
Not even in the case of simple information extraction from a text you can ignore hallucinations, because if you provide a list of names and ask for all those starting with “A” you cannot be certain that all names output will actually start with “A” and most certainly cannot be certain that all correct names will be in the output. And this behavior cannot (as of today) be corrected on the LLM we have right now (the first part yes, the second part no).
So, LLM with hallucinations are a very powerful tool, but not the tools they are being sold as.