GPT-3 has no idea what it’s talking about

The "Chinese room argument" says that external behaviour cannot be proof of consciousness, intelligence, understanding etc. What my comment above says is that you can't explain a word A by means of another word, B, unless you already know the meaning of B.

My comment is relevant to the question of whether GPT-3 has "understanding" or not, because in order for GPT-3 to understand the meaning of a word A in terms of a meaning of a word B, it needs to already know the meaning of the word B. However, this is what we wish to know, whether GPT-3 knows the meaning of any word. Observing that GPT-3 can use a new word in the place of a different word doesn't tell us whether it knows the meaning of the original word.

As of yet, no, there is no formal test that would convince me or a majority of reserachers in AI that a model "knows", "understands" or anything like that. The reason is not that I am too stubborn, say. Rather there simply aren't such tests available yet. One reason for that is that we don't, well, understand what it means to "understand". We don't have a commonly accepted formal definition of such ability. Without that, we can't really design tests to prove that some system has it.

The take away is that it will be a long time before we can know for sure that a system is displaying intelligence, understanding, etc. This may be unsatisfying- but the alternative is to design meaningless tests that prove not what we are trying to prove and proclaim the goal proven if the tests pass. This does not go well with the purpose of scientific endeavour, which is to acquire knowledge- not pass tests and make big proclamations about winning this or that competition.

In short, I'm not saying that computers can't have understanding, or that we can't know if they do. I'm saying that right now, these things are not possible, with current technology.

It's an activation pattern, but it's not "just an activation pattern". The face activation relies on previous layers, the detection of each component. We can conjecture probably fair to say human brain object recognition (and other subconscious processes) use similar principles. All the components required to efficiently "understand", say a face, are there (not the shown visualization and architecture is for AlexNet, by now a very old and primitive model). I don't think we can ask for much more.

What transformers do differently from CNNs is attention/recurrence. They have modifiable internal state, while feedforward models just have the feedforward state that can't temporally be reused (which is what we mean by 'algorithm'). This is a feature of logical thinking (and our own logical thinking), but I suspect most of what is meant by understanding the world is already contained in the internal structure captured by those models. Most of I understanding, as far as I can tell, comes from both this structural, intuitive inference (that CNNs and language models do), allied with our ability to think -- that is, talk to ourselves -- and thus build explanations and models on the fly, still reliant on the structural, intuitive understanding that comes from just very large networks generating abstract representations, classifications, etc.