I find myself missing the golden age of Google, where it actually returned results and answers that you wanted, on subjects you were looking for. Even now, versus two years ago, I tried finding a snippet of a notice in a newspaper with specifics about a name change someone petitioned in California. I found it then, and had bookmarked it, but trying to find it again just turns up absolute garbage, thankfully I found t…
I concur. Intelligence does not come from the transformer architecture, or any specific of the model. It comes from the language corpus. Human intelligence too, except for physical stuff. All our advanced skills come from language.
You take 300GB of text and put it through a randomly initialised transformer and you get chatGPT. You immerse a baby in human language, it becomes a modern adult, with all our abilities. Without language, and that includes other humans and tech, we'd be just weaker apes.