AI is just someone else's intelligence

I feel that the big hidden implication that these kinds of articles are trying to say is "AI is not real intelligence", further implying something along the lines of "AI will never be conscious" (as it's hard to come up with another definition of "real" intelligence except the-human-kind).

I'd like to propose a counterargument:

Assumptions: Theory of evolution is true. The primordial single-cell organism from which we all evolved was not conscious, but rather just a biological machine. Humans are conscious.

Deduction: Somewhere along the line between the primordial single-cell organism and a human being, there was a transition from non-consciousness to consciousness, and the only [*] factor in that transition was the complexity of the nervous system.

Conclusion: Consciousness (or, "real" intelligence) arises from the complexity of a machine. AI can, in principle, become conscious.

Yes, we know how AI works, because we built it. But why would that make consciousness arising from a sufficiently-complex statistical model impossible?

[*] as per apendleton's comment, I have made mistake here: complexity is not the only factor, but is a necessary one in creation of consciousness.

I like that articulation. But I don't see why emergent intelligence has to even necessarily be in the mix for an ML / AI tool to be as helpful / insightful as the current systems seem to be. At least based on how they've felt in everyday use in my development process at work and as a consumer etc. Theyre about as helpful as Id expect an off-brand all-of-the-internet philosopher stone chatbot assistant to be.

It communicates an intelligence, but it's someone else's intelligence, not the book's.

> the only factor in that transition was the complexity of the nervous system

It seems likely to me that some arrangement of nerves is possible that's comparably complex to ours, but does not produce consciousness. (I dunno, maybe some organism with much more complex sensory organs than ours that devotes so much complexity budget to that that it only has enough left to devote to general cognition to give it the intelligence of a mushroom, who knows). In other words: I suspect complexity is necessary but not sufficient for consciousness to occur. I don't think that takes away from your suggestion that consciousness in AI systems is _possible_, but I don't think it's the case that it's an inevitable outcome if only we can make our systems sufficiently complex. There's probably something about the specific structure of the complex thing we'll need to master as well.

This logic only applies to generative pre-training, behavior cloning, and other training methods which rely on learning to mimic well-structured content from the real world.

It does not apply to intelligence gathered through methods like RL.

How does the author think about the intelligence of AlphaGo, for instance, which was trained entirely by self-play?