Large models of what? Mistaking engineering achievements for linguistic agency

Yes. Obviously. I can create plans and think on them. I can think without the need for an internal monologue or talking to myself. This has nothing to do with modalities either: I do not think through text. I use text when I do as a way of conveying the thoughts I already have.

Anyone who claims we don't have the ability to form concepts in our head distinct from the medium in which they're transmitted is saying we're effectively ears, eyes, and skin. That the modality is what is important for intelligence.

This is clearly false, and academic silliness aside, yes—emphatically—humans and intelligent agents have internal concepts and models of the world.

The "generation" of strings is related to this output format. It's critical to how they work. Legerdemain has been performed to argue that that's irrelevant, and the real intelligence or concepts are sitting inside the network architecture of the trained model prior to generation. But if that were the case, generation could be done based upon the conceptual representation, and not a syntactical representation token by token. This is not currently the case with LLMs.

I'd turn this question around: if my question is irrelevant, how would one go about building an effective real-world LLM that understand concepts and doesn't use likelihood lookups on a token-per-token basis, but instead generated directly from the conceptual basis? Such an argument, if it exists, would make me very happy.

Please note, I understand that there are prior systems which do this. Generative zero-shot transformer models didn't adopt this approach because it's elegant, but because it is efficient to compute with large data sets and has useful efficacy in generating strings. Some are creative. Some are more "accurate." The temperature parameter can affect which of those cases it selects, if any.

People have short memories, but the people who are both appreciative of LLMs as an engineering feat and critical of their claims of intelligence have been saying this for years. They've said that their token likelihood model is effective for seeing things well covered in the data set. They've been saying that due to the sparsity and structure of human language, large scale approximate compression ("curve fits") would be highly effective and efficient. They've been saying that due to the fact these are large scale fits of a data set, the models would eventually converge to something looking like the known knowledge they're trained on, and not exponentially accelerate in knowledge. All of these predictions have proved to be correct or looking highly likely at this point.

Transformer-based LLMs are a neat algorithmic approach to curve fits. But they are curve fits. Things like cosine transforms in JPEGs, wavelet or Fourier reconstruction in CAT scans, audio signal reconstruction from basis functions are also approximate reconstruction models that function along these lines, albeit in a nice Euclidian space without the generative parts of a transformer. But it was precisely knowledge of how systems like that worked which allowed scientists to understand and predict the limitations of these systems a long time ago. Lots of money and fresh eyes have created a useful computation technique, but these insights have been forgotten. I hope—truly—that progress happens in this space. But the critiques stand and there would be lots to gain by a less-hyped acknowledgement of where we are with these models and the tradeoffs baked into them as a compromise for them to be useful.

> The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide.

Regardless of whether this is true for existing transformer-based models, this is not true for all computable conditional probability distributions over text.

Any computable task can be framed as sampling from some conditional probability distribution. (If the task is deterministic, that just means that the conditional probability distribution to sample from is one which has probability 1 for some string, when conditioned on the thing it is to be conditioned on.)

Whether transformer based models are lookup tables or not, not all computable probability distributions over text are. (As, of course, not all computable tasks can be expressed as a simple finite lookup table.)

I don’t know exactly what you mean by “generally understand concepts”, though I suppose

> They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system.

is describing that somewhat. And, in that case, if there is any computational process which counts as having “enough conceptual power to generate new facts and know that they are facts consistent with the model”, then, a computable conditional probability distribution over strings conditioned on their prefixes, and therefore also a computable probability distribution over next tokens given all-tokens-so-far , is also (theoretically) capable of that.

And so, it would follow that “it only predicts the next token” doesn’t (in principle/theory) preclude it having such an understanding of concepts, unless no computational process ever can.

What probably happened, and still happens, is that some people underestimate the intelligence of other people and think they're not as intelligent as themselves, not that they don't have intelligence.