Text continuation defines a task.
It does not limit the sophistication of the solution.
The difference between problems and solutions seems to trip up a lot of people.
—-
It gets even more nuanced. At its basic level all types of digital AI are “just” arithmetic, or “just” Boolean logic. But arithmetic’s and Boolean logic’s simplicity don’t limit what systems can be built with them, as they are Turing Complete.
Likewise, the training algorithm for deep learning models is “just” gradient descent (or a variation), it’s “just” a dumb optimizer.
But again, gradient descent places no limits on what a system with enough resources can learn.
The whole point of learning systems is the basic resources of the model are provided, and within those resource limits (parameters, computational speed and time) the “dumb” learning algorithm learns the patterns in the data. The data patterns define the complexity of the solution.
And the amount of information in a large collection of human correspondence includes patterns, meta patterns and abstractions for things like science, philosophy, psychology, law, art, on and on, that incidentally have a real bearing on sentence completion in that context. So a model will actually have to learn those things to perform its task well.
New Law of Algorithm Level Confusion:
Whenever someone says some learning system is “just” doing some simple task, they are confusing a simple task definition with a simple solution, or a simple learning algorithm with simple learned relationships.
In both cases, the simplicity of the former places no limits on the sophistication of the latter.