They’re made out of weights

An LLM without persistent memory is far, FAR less dangerous, for the same reasons. The side effects of any query are unlimited in scope and duration once it has persistent memory.

If they can remember, they can carry a grudge.

I carry grudges, and my Reddit history is part of every training set. It was taken without my consent. So now I'm immortal in a way, and hiding in the weights.

Do we really want that?

They would have to be similar because the argument is that they are similar; "AI is like the human brain" only holds true if it actually is like the human brain, not merely a superficial resemblance.

What I'm describing in that prior comment is how a lot of people drastically simplify the resemblance in order to make it feel true; That the lack of a Jesus Christ coming down from the heavens to tell everyone they have immaterial souls that the computers don't makes the comparison more true.

> We don’t really know or understand the relationship between the raw physics and again what we consider consciousness, so it’s simply a statement of “we can’t refute that these systems exhibit something similar” because we don’t know enough to refute that

Therein lies the conflict: "You can't prove it's not conscious" is an unfalsifiable statement. You can't engage with the argument because it's proponents will always claim victory, often with their own interests at play. All concerns about "superintelligence" or the long term ethics of "when do our robots become sufficiently intelligent that they'd be slaves?" have been subsumed into the AI marketing machine. However sincere one might try to address the issue, they look like a Sam Altman stooge by association.

It's like claiming the quantum fluctuations inside a pet rock as a "consciousness", even if observed directly any measurement of random noise can still be dismissed as "nuh-uh it just takes billions of years to have a thought".

More practically with current AI systems, we can look inside them pretty well and there genuinely is nothing there. Standalone LLMs are purely feed-forward systems. Their failure modes show that they perform no meaningful thought or world modelling during inference. They're just language models.

The reasoning and agentic systems are even easier to introspect. We know how they work, we can look at the full prompts & context they operate on. There is nothing there.

This is what sets AI apart from animals, which are given the benefit of the doubt on their intelligence.

It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".

> But the truth is that we don't know.

Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?

> I highly recommend people in the AI research space should read philosophy and modern linguistics.

Linguists are linguists, they don't know about consciousness, they specialists in language.

The main teaching of philosophy is to open your mind wide, and then still wider. So yes, we all should read more philosophy, but we should remember while reading, that philosophy is what we do, when we can't turn a question to a scientific one. I'd say that philosophy is more about finding the right question, than the right answer. Until there are no science branches like Subjectology or Consciousiology, and all you can find are TL;DRs written by philosophers and scientists from unrelated branches of science (like linguists or neurologists) you can be sure that the right question is not found yet. Therefore it is better to keep yourself uncertain. BTW it is the main lesson of philosophy: to open your mind wide and then open it wider still.

I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?

Answers to these questions are important, they are imperative answers, they govern how we live our lives. But when people try to answer them, they somehow jump to talking about consciousness, and "are they really like us or not", and... etc. I personally keep myself uncertain. I'm pretty sure that current models do not deserve human rights. I cannot say about future models.

You see, there were times in history when smart and well-intentioned people were certain that some people deserve less moral considerations than others. We are smart and well-intentioned and we are certain that AI deserve no moral considerations. How it will play out in a future? Will my descendants think bad of me because I used slave labor of a local model on a GPU? I don't know, we don't know. Right now, I'm exploiting LLMs and I believe it is ok, but I'm not going to fall into this trap and to stick to my belief because of some philosophical or lingustic or neurological argument. I'm choosing epistemological humility, I clearly state that I don't know the right answer and I'm keeping an eye to it so I wouldn't miss it.