Ask HN: Why does no one seem to care that AI gives wrong answers?

Lots of people care.

From a coding perspective, proper technical systems already have checks and balances (e.g. test cases) to catch bad code, and is something that's important to have regardless of generative AI usage.

From a creative/informational perspective, there are stories every day of hallucinations and the tech companies are correctly dunked on because of it. That's more product management error than AI error.

AI hallucination isn't a showstopper issue, it just has to be worked around.

Blind optimism

So, if you dealt with a person who knew all the vocabulary related to a field, and could make well-constructed sentences about that field, and sounded confident, it would almost always mean they had spent a lot of time studying that field. That tends to mean that, although they may occasionally make a mistake, they will usually be correct. People apply the same intuition to LLMs, and because it's not a person (and it's not intelligent), this intuition is way off.

There is, additionally, the fact that there is no easy (or even medium difficult) way to fix this aspect of LLM's, and it means that the choices are either: 1) ship it now anyway and hope people pay for it regardless 2) admit that this is a niche product, useful in certain situations but not for most

Option 1 means you get a lot of money (at least for a little while). Option 2 doesn't.

(1) Some problems are probabilistic either in theory or practice. For instance there could be a sentiment analysis problem where the state of the art was 67% accuracy 5 years ago and with an LLM it is easy to get 85% accuracy. 100% accuracy is going to be impossible anyway because sometimes you really can't figure how somebody feels.

(2) It's a big topic that could be addressed in different ways but I'll boil it down to "people are sloppy" and that many people become uncomfortable with complex problems that have high stakes answers and will trade correctness for good vibes.

(3) LLMs are good at seducing people. To take an example, I know that I was born the same day as a famous baseball player who was also born exactly a year before an even more famous cricket player. I tried to get Microsoft's Copilot to recognize this situation but it struggled, thinking they were born on the same day or a day apart rather than a whole year. Once I laid it out explicitly and my own personal connection it had effusive praise and said I must be really happy to be connected to some sports legends like that, which I am. That kind of praise works on people.

(4) A lot of people think that fixing LLMs is going to be easy. For instance I'll point out that Copilot is completely unable to put items in orders that aren't excessively easy (like US states in reverse alphabetical order) and others will point out that Copilot could just write a Python program that does the sorting.

That's right and it is part of the answer, but it just puts off the problem. What's really irksome about Copilot's inability to sort is that it doesn't know that it can't sort, if you ask it what the probability is that it will sort a list in the right order it will tell you that it is very high. It's not so easy to know what is possible in terms of algorithms either, see

https://en.wikipedia.org/wiki/Collatz_conjecture

as evidence that it's (practically) impossible to completely understand very simple programs. See the book

https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach

for interesting meditations on what a chatbot can and can't do. My take is that LLMs as we know them will reach an asymptote and not improve explosively with more investment, but who knows?

There's no "hallucination". This word is simply PR spin to disguise faulty output.