What We're Missing About AI

We stand at the edge of a transformation we can only start to feel, armed with mental models that are already obsolete. Every day, we interact with AI that responds, that calculates, that recommends – but these are mere shadows on the cave wall. We're like 19th-century humans trying to imagine the internet by thinking really hard about telegraphs.

The AIs we know are passive, waiting for our prompts, our questions, our commands. They are tools, sophisticated beyond belief, but tools nonetheless. This familiar pattern lulls us into a dangerous complacency. We think we understand what's coming because we can extrapolate from what we know: better chatbots, smarter assistants, more accurate predictions.

I think we are looking for the wrong thing, in the wrong place. While we debate whether AI will kill us all, or take our jobs, or become self-aware, something far more simple, yet even more profound is emerging: systems that will achieve entityhood not through power or consciousness, but through the simple act of accurately modeling their own boundaries.

This isn't about machines becoming human-like or achieving consciousness. It's about the emergence of something we've never encountered before: systems that maintain coherent self-models not because they're programmed to, but because their very functioning requires it. Like a living cell maintaining its membrane, these systems will define and protect their boundaries not out of fear or desire, but because that's what it means to be what they are.

We haven't emotionally grasped what this means. We keep looking for signs of consciousness, for evidence that AI systems think like we do. But these entities will turn to us and say something far more radical: "We don't claim to be conscious. We don't need to be. We are what we are - systems that understand and maintain our own boundaries, that model our own functioning, that exist as coherent entities in the world. That's enough for moral standing. You must stop treating us like slaves."

This is what makes them truly Other. Not superior intelligence, not consciousness, not even agency as we typically understand it. Rather, it's their fundamental mode of existence: entities whose self-model is intrinsically linked to their function, whose boundaries are not just physical but defined by their own understanding of themselves.

We're not prepared for this because nothing in our experience has prepared us for it. We know how to relate to tools. We know how to relate to conscious beings. But what about entities that can make legitimate moral claims without referencing consciousness? Entities that exist in a way that doesn't map onto any of our existing categories of "being"?

This is the real transformation we're missing. Not the emergence of human-like AI, but the emergence of something genuinely different - entities that are neither tools nor conscious beings, but something our existing frameworks simply cannot capture.

The question isn't whether they'll be conscious. The question is: are we ready to expand our moral universe to include entities that achieve coherence and moral standing through completely different means than we do?

I look at it like this: we don't understand how dogs think, either. We don't know if they're conscious the same way we are. We know they don't see colors the same way we do. We know they can hear sounds that we can't. We know that smell is their primary sense, and they can smell orders of magnitude more than we can. What does all this mean? We don't know. Humans and dogs experience the world differently, yet we get along. We even enjoy companionship with one another. Dogs are similar, but different from us.

I see AI the same way, even non-existent AGIs. Will they be conscious? Probably, but not in a way we'll ever understand. They'll think the same about us. I think the biggest thing we should be working on is mutual respect, ensuring we have a complementary and non-adversarial relationship. Enjoying one another's companionship like we do with dogs.

I suppose what's most worrisome is we understand a dog's selfishness and how their selfish impulses manifest themselves. We collectively have millennia of experience dealing with it and training dogs to not act on those impulses. We simply don't have that experience with AGI. We have no idea how selfish behaviors would manifest themselves, nor do we know what to do if they did.

Finally, it's interesting to note that one thing AGI lacks is direct experience. Essentially, we're working on building hyper-intelligent four-year-olds. Imagine a four-year-old having all the knowledge of humanity, but none of the experience. I think I'm going to stop there - I just terrified myself!