The single biggest annoyance with Opus 5 is that it writes too elliptically. Sentences that orbit a point, then jump to it like it's a revealed insight. Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end. It is definitely mo…
Pondering this one night last week, I realized that because LLMs can only reason with written language, what we might be seeing emerge with Opus’s load-bearing mumbo jumbo is its own creole for structural reasoning. Not only are our brains wide, our senses are, too. I slow down to a crawl when I have to read actual math in a CS paper, but show me diagrams and I can reason about whatever sort of data structure or algo…
Is that right? Think of code that draws a square, the representation of that code in storage, the movement of electrons, and the 'actual' square on the screen -- these are all to us transcriptions of same thing across different domains or media, and we can deterministically translate back and forth between them, but there's no real, actual, essential identity property. The code isn't the square. The electricity isn't the code. The storage isn't the electricity.
I think of LLM reasoning and output similarly. The underlying graph of weights, matrices, and other data aren't knowledge, understanding, or language. 'Translating' the system's output to language is jusas valid and correct as translating it into some visual representation that would be incoherent to us, like a sequence of flashing lights or imperceptible noise patterns projected over an image of a dog.
I guess this is all a very long winded way of restating Chinese-room problem: we feed the man in the room a message; he returns one that, for all the world, is indistinguishable from a "real" response that you and I might send, but, like you said, he has no access to sense data. He also has no access to the biology underlying real mental processes. He also doesn't have any personhood that we can discern. He has, rather, gradually developed through reinforcement the tendency to provide responses approximating all of all of that.
I'm not sure the epistemological question "does he understand" (which is what the Chinese-room problem asks) has any meaningful answer. There's no mind, so there's no understanding. What there is, rather, is a system that generate patterns that we map to language and that our brains therefore map to communication, personhood, meaning, etc. It's the square I mentioned earlier. It's to us a convincing simulacrum, and it may be faithful enough to us to stand in those things, but that's not what it is.
My sense is that LLMs are (a) the big-data Pyramids of Giza and (b) a consequence of hardware and software developing ways of generating abstractions that capture and generate more complex patterns than were previously possible to capture or generate in a manner comprehensible to humans. Everything humans do follows some kind of pattern. Language is the perfect way for a machine to capture that, because is simpler than the world itself and has clear rules and patterns, encoded in representations computers already have, that, modeled well enough, can generate output indistinguishable from the real thing -- what you or I might do with it.
But the pattern matching that it does, and that we translate into language, is much more numerically rigorous and complex than anything you or I consciously do with words (hell, most people can't even figure out when to use "lie" vs "lay") and not doing what you or I do with it. It's not language. It's the square on the screen.