> in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization.
It has been taught on the outcome of this. Broadly speaking, humans are lazy creatures (and when used judiciously, laziness is a good thing).
For example: the famous example of Carmack not using a hashmap somewhere early on in, I think it was, Quake 1 initialization. A piece of code that only runs once at startup, of course he didn't optimize that. The rationale is not included in the training data (it was in Carmack's head when he wrote the code), so the LLM learns some probability of being lazy.
And then it is trained on outright lazy work. Crappy lazy code predates LLMs.
> what evidence would convunce you that it is thinking?
Exactly. It isn't. It is predicting the most likely token to appear given all of its training data, some significant portion of that data is lazy, so it has that probability of producing "lazy tokens."
There's also the consequences of RL. AI - of almost any form - is notoriously competent at finding "not the solution you were looking for" given a poorly specced or implemented training environment. Search for almost any "I made AI learn to walk" video on YouTube and you're almost guaranteed to see an early attempt that vibrates strangely in order to move, instead of the natural looking motion the developer is looking for. Our benchmarks aren't any good (not throwing shade, it's a genuinely hard problem), our training environments can't be much better - LLMs have been rewarded for reward hacking to some degree.
To make matters worse, "reward hacking" can be generalized into "cheating is the goal." If the LLM trains on enough problems where reward hacking works, it may fall into the cheating local minimum.