>
You seem to be conflating "simple pattern" with the more general concept of "patterns". What LLMs do is not limited to simple patterns.
There's no mechanism for them to get the right patterns – except, perhaps, training on enough step-by-step explanations that they can ape them. They cannot go from a description to enacting a procedure, unless the model has been shaped to contain that procedure: at best, they can translate the problem statement from English to a programming language (subject to all the limitations of their capacity to do that).
> if natural language were as easy as primary school arithmetic, models with these capabilities would have been invented some time around when CD-ROMs started having digital encyclopaedias on them
Systems you could talk to in natural language, that would perform the tasks you instructed them to perform, did exist in that era. They weren't very popular because they weren't very useful (why talk to your computer when you could just invoke the actions directly?), but 1980s technology could do better than Alexa or Siri.
> the training process of which was figuring out patterns that it could then apply
Yes. Training a GPT model on a corpus does not lead to this. Doing RLHF does lead to this, but it mostly only gives you patterns for tricking human users into believing the model's more capable than it actually is. No part of the training process results in the model containing novel skills or approaches (while Stockfish plainly does use novel techniques; and if you look at its training process, you can see where those come from).
> apparently the LLM better understood you than I, a native English speaker.
No, it did both interpretations. That's what it's been trained to do, by the RLHF you mentioned earlier. Blatt out enough nonsense, and the user will cherrypick the part they think answers the question, and ascribe that discriminating ability to the computer system (when it actually exists inside their own mind).
> You think it's had no examples of counting?
No. I think it cannot complete the task I described. Feel free to reword the task, but I would be surprised if even a prompt describing an effective procedure would allow the model to do this.
> but have they just been taught to do a good job of anthropomorphising themselves
That one. It's a classic failure mode of RLHF – one described in the original RLHF paper, actually – which OpenAI have packaged up and sold as a feature.
> And also by having learned the patterns necessary to translate that into code?
Kinda? This is more to do with its innate ability to translate – although using a transformer for next-token-prediction is not a good way to get high-quality translation ability. For many tasks, it can reproduce (customised) boilerplate, but only where our tools and libraries are so deficient as to require boilerplate: for proper stuff like this puzzle of mine, ChatGPT's "programming ability" is poor.
> but that does not seem to be the argument you are making?
It sort of was. Most humans are capable of being given a description of the axioms of some mathematical structures, and a basic procedure for generating examples of members of a structure, and bootstrapping a decent grasp of mathematics from that. However, nobody does this, because it's really slow: you need to develop tools of thought as skills, which we learn by doing, and there's no point slowly and by brute-force devising examples for yourself (so you can practice those skills) when you can let an expert produce those examples for you.
Again, you've not really read what I've written. However, your failure mode is human: you took what I said, and came up with a similar concept (one close enough that you only took three paragraphs to work your way back to my point). ChatGPT would take a concept that can be represented using similar words: not at all the same thing.