Earlier quoted context omitted.
Funnily enough my bachelors thesis was “Wittgensteinian problems for Artificial General Intelligence” so I also have grounds for being bearish, even though I respect what is happening at the moment and have hope for the future.
Is the Wittgensteinian problem that the AGI wouldn't understand concepts like we do, or is the problem that it wouldn't be able to use them better than a human?
One of them is that we only have language with which to build an AI - language and symbols. Yet these nevertheless capture absolutely nothing about what it is to perceive and understand, on a human level, so yeah. It’s kind of what Searle was getting at - symbolic outputs and translation based on rules doesn’t get you intelligence, even if it produces a system that looks as if it is intelligent.
There’s also that we have an intersubjective attunement to others that is pre-linguistic. Ultimately “meaning is use” which is a pithy take on later Wittgenstein- there’s nothing in language which means anything at all. So it’s not that it wouldn’t be able to use language or concepts better than we do, it’s that we change with the AI and don’t accept them - particularly as they start to become uncanny or unpredictable.
There really is something to sentience that we all know that cannot be reduced to language or machinery.
Our language about machinery, and intelligent machinery, is full of usage mistakes and categorical errors that we overlook, but which are more serious when analysed.
We are complete beings - my intelligence is part of a system, and it makes no sense to abstract it out of that system in order to nail it down.
“Pain” cannot be reduced down to or captured in language. Suffering is part of our intelligence and we’d be nowhere without it.
So ideas kind of like that and more.
AGI would have to suffer to become general enough to warrant the use of the word general.