Earlier quoted context omitted.
I think it has to do with the training regime and fixed-computation time nature of feedforward neural networks. Recurrent neural networks have the recursion as part of the training regime . GPT only has auto-regressive "recursion" as part of the inference runtime regime. I think Hofstadter is surprised that you can appear so intelligent without any recursion in the learning/training regime, with the added implication…
That seems to correspond to the "is or isn't it feed forward" debate going on here, so I guess I see where the confusion comes from (both for him and in terms of calling LLMs FF/not FF)
Consider a simpler case: a small neural network that takes 2 numbers and adds them together, producing 1 number as output.
This network is very obviously feedforward, and probably very tiny with few layers.
Say I have a list of numbers [1, 2, 5] that I want to sum. If I send 1 and 2 through the network, get 3 as a result, then I send 3 and 5 through the network, and get a final answer of 8, my network has not suddenly become non-feedforward just because I fed the output back into it.
The key distinguishing factor between feedforward and non-feedforward is if the network itself loops back around and, at training time, it learns how to make use of this ability to pass data to itself to maintain some hidden context between passes.
There is no such learned hidden context in my addition example, and none in GPT.
---
There actually is a tiny caveat here: the RL fine-tuning process OpenAI has done on its models ("RLHF" & friends) actually does allow for a very, very small amount of information leakage between passes because you are rewarding whole responses, so the model can learn little patterns of what tokens in the beginning of the response led to certain tokens at the end of the response and reinforce those patterns.
The model could learn to encode small bits of "hidden" information in particular token choices that the human raters wouldn't notice. In this case, there is a (small but non-zero) amount of learned hidden context. But this is not what Hofstadter is talking about -- the non-RLHF'd base model is just as intelligent, just harder to use.