Some thoughts on autoregressive models

You seem to continue to make a lot of claims without basis.

> they purely rely on pattern matching

Yes.

> because they don't have the internal representation of what they really deal with (algebra)

Wait what? No. You can't claim this yet as it's an open question. It may well be the case they do have an internal representation of algebra, and even a world model for that matter, if flawed.

I think you need to be more aware of the current research in LLM interpretability. The answers to these questions are hardly as definitive as you make it seem.

I do actually read a lot about LLM interpretability, and this is my own conclusion (should've phrased it like "they don't seem to have"). I do actually consider this an open question, so I'm a bit confused as to why you think this way - perhaps due to my phrasing (I just had a very long flight), but know that is not the case and I always doubt things. In fact I said after the text you quoted that the question is actually quite open (mentioning reasoning models, but to be honest, it's not exclusive to them, just more apparent in some ways).

I might also clarify (here and probably in my article when I have the time to do so). LLMs "do" build internal models in the sense that, at the same time:

- They organize knowledge by domain in a unified network

- They're capable of generalization (already mentioned and acknowledged at the very beginning of the article)

However these models, while they share parallels with human cognition, lack substance and can't replicate (yet) the deep integrated cognitive model of humans. That is where current interpretability research is at, and probably SOTA LLMs too. My own opinion and speculation is that autoregressive models will never get to a satisfying approximation level of the human-level cognition since humans' thinking process seems to be more than autoregressive components, aligning with current psychology. But that doesn't mean architectures won't evolve.

Do not misunderstand that because I said they're pattern matching machines, that they will be unable to properly "think". In fact, the line between pattern matching and thinking is actually quite blurry.

> We can't interpret what actually happens when the LLM is actually going through the process of generating a response.

There are 2 things we’re talking about here.

There’s the physical, mechanical operations going on during inference and there’s potentially a higher order process happening as an emergent property of those mechanical operations.

We know precisely the mechanical operations that take place during inference as they are machine instructions which are both man-made and very well understood. I hope we can agree here.

Then there’s potentially a higher order process. The existence of that process and what that process is still a mystery.

We do not know how the human brain works, physically. We can’t inspect discrete units of brain operations as we can with machine instructions.

For that reason, it is uncritical to assume that there is any kind of “thought” process occurring at inference which is similar to our thought processes.

Comparing the two is like apples and oranges anyway and is pedantic in a non-useful way, especially with our limited understanding of the human brain.

> There are 2 things we’re talking about here.

I was never actually talking about the physical mechanisms. Sure we can agree that GPUs, logical gates, etc physically work in a certain way. That just isn't important here at all.

> For that reason, it is uncritical to assume that there is any kind of “thought” process occurring at inference which is similar to our thought processes.

I wasn't intending to raise concerns over emergent consciousness or similar. Whether thought goes on is a bit less clear depending on how you define thought, but that still wasn't the point I was making.

We have effectively abandoned the alignment problem and the interoperability problem. Sure we know how GPUs work, and we don't need to assume that consciousness emerged, but we don't know why the model gives a certain answer. We're empowering these models with more and more authority, not only are they given access to the public internet but now we're making agents that are starting to interact with the world on our behalf. Models are given plenty of resources and access to do very dangerous things if they tried to, and my point is we don't have any idea what goes on other than input/output pairs. There's a lot of risk there.

> Comparing the two is like apples and oranges anyway and is pedantic in a non-useful way, especially with our limited understanding of the human brain.

Comparing the two is precisely what we're meant to do. If the comparison wasn't intended they wouldn't be called "artificial intelligence". That isn't pedantic, if the term isn't meant to imply the comparison then they were either accidentally or intentionally named horribly.

Haha, well its funny sometimes when you realize too late there were two different conversations happening.

I definitely agree on the term machine learning - it seems a much better fit but still doesn't feel quite right. Naming things is hard, but AI seems particularly egregious here.

> The fact that the underlying processes are well understood means that, if we so wished, we could work backwards and understand what the model is doing.

I'm not sure we can take that leap. We understand pretty well how a neuron functions but we understand very little about how the brain works or how it relates to what we experience. We understand how light is initially recognized in the eye with cones and rods, but we don't really know exactly how it goes from there to what we experience as vision.

In complex systems its often easy to understand the function of a small, more fundamental but of the system. Its much harder to understand the full system, and if you do you should be able to predict it. For LLMs, that would mean they could predict a model's output for a given input (even if that prediction has to account to randomness added into the inference algorithm).