Is AI reasoning right for the wrong reasons?

LLMs lack qualia, among other things.

If I ask an LLM "what is an apple?" it tells me:

> An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks.

If I ask an LLM "what is a mundu fruit?" it tells me:

> Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia, Malaysia, Thailand, and Cambodia. It comes from a small evergreen tree in the same genus as mangosteen.

I've never eaten a mundu fruit. To me, an apple and a mundu fruit are categorically different. An apple is a fruit that I've held, touched, tasted, eaten, enjoyed, cooked with. A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.

"Existing in an abstract way" is how an LLM experiences everything. To an LLM, an apple and a mundu fruit are in the same category. The LLM has been trained on text about both fruit, it's seen images of both fruit, it knows everything that has been recorded about both fruit ...except everything that's important to know about a fruit.

Many of our issues with LLMs arise because from the LLM's perspective, nothing exists. If Claude accidentally deletes your production database, it may well apologize afterward, but only because an apology is statistically likely. It doesn't feel guilt like a human would, and the lack of consequences makes any action an LLM takes inherently frivolous. We want them to understand what's real and what's not, but without any lived experience perhaps that's an unreasonable expectation.

Not only that! LLM can also learn how to repeatedly apply f() for certain arguments, and run the shortcut. Sometimes, the shortcut learned is not actually repeated application of f(), which breaks semantic soundness of LLM's reasoning chain.

These shortcuts can greatly help reasoning, but they are not guaranteed to be sound. So yeah, in that case, LLMs are reasoning right (using shortcuts) for the wrong reasons (learning them from data instead by running actual repeated application and memoizing the resulting rule).

> We want them to understand what's

We want them to assess what's true and what's not.

Holy run-on sentence batman, maybe you do need AI.

Your employers are (likely, some assumption here) paying for the productivity of a full time employee. You have figured out that they're satisfied with a third of that productivity.

You are the reason why there are so many return-to-office mandates, because you're projecting the idea that remote employees cannot be trusted. You're making it worse for everyone. Good luck.

Edit, added after the original reply: You are also taking 3 jobs in a market where many people struggle to find one. You are hoarding resources. Enjoy the 3 paychecks, also rot in hell.