Earlier quoted context omitted.
> Do you really think consciousness lives in weights and an input vector? So far as we can tell, all physics, and hence all chemistry, and hence all biology, and hence all brain function, and hence consciousness, can be expressed as the weights of some matrix and input vector. We don't know which bits of the matrix for the whole human body are the ones which give rise to qualia. We don't know what the minimum represe…
You're assertion that consciousness, chemistry, and biology can be reduced to matrix computations requires justification. For one, chemistry, biology, and physics are models of reality. Secondly, reality is far, far messier and more continuous than discrete computational steps that are rountripped. Neural nets seem far too static to simulate consciousness properly. Even the largest LLMs today have fewer active comput…
https://en.wikipedia.org/wiki/Matrix_mechanics
"It matches reality to the limits we can test it" is the necessary and sufficient justification.
> For one, chemistry, biology, and physics are models of reality.
Yes. And?
The only reason we know that QM and GR are not both true is that they're incompatible, no observation we have been able to make to date (so far as I know) contradicts either of them.
> Secondly, reality is far, far messier and more continuous than discrete computational steps that are rountripped.
It will be delightful and surprising if consciousness is hiding in the 128th bit of binary representations of floating point numbers. Like finding a message from god (any god) in the digits of π well before expected by the necessary behaviour of transcendental numbers.
> Neural nets seem far too static to simulate consciousness properly. Even the largest LLMs today have fewer active computational units than the number of neurons in a few square inches of cortex.
Until we know what consciousness is at a mechanistic level, we don't know what the minimum is to get it, and we don't know how its nature changes as it gets more complex. What's the smallest agglomeration of H2O molecules that counts as "wet"? Even a fluid dynamics simulation on a square grid of a few hundred cells on each side will show turbulence.
Lots of open questions, but they're so open we can't even rule out the floor as yet.
> but the first round of AGI won't be close.
Every letter means a different thing to each responder, they're not really boolean though they're often discussed that way, and the whole is often used to mean something not implied by the parts.
It is perfectly reasonable use of each initial in "AGI" to say that even the first InstructGPT model (predecessor to ChatGPT) is "an AGI": it is a general purpose artificial intelligence, as per the standard academic use of "artificial intelligence".