>Again, I've had this exact experience with people many times as well
There's a difference between some people sometimes needing to be reminded to do something, and them flat out not being able to do it due to fundamental cognitive limitations.
>"E.g. I just tested by asking GPT4 for a multiplication method and then asked it to use it on two numbers I provided and show its working, and it did just fine."
That's because GPT4 has been custom tuned and trained on that specific task as well, along with many others. It's that training, why it was necessary and how it works that the paper referred to previously was about.
This is literally the subject under discussion. You're using the fact the system was custom trained to do that specific task as is therefore good at it, along with other basic mathematical tasks, as evidence that it has a general ability that doesn't need to be custom trained.
This is what I'm talking about and as these models get better specific problem domain training two things will happen. One is that they will become dramatically more powerful and capable tools. That's great.
The other is that more and more people will come to fundamentally misunderstand how they function and what they do, because they will see that they produce similar results to humans in many ways. They will infer from that they work cognitively in the same way as humans, and will reason about their abilities on that basis and will make errors as a result, because they're not aware of the highly specialist training the systems had to go through precisely because there are very important and impactful ways in which they don't cognitively function like humans.
This matters, particularly when non specialists like politicians are making decisions about how this technology should be regulated and governed.