A warning about 'model welfare'

I mean, if it's a cow, you can get away with this. The cows aren't going to rise up against you. Well, they might take out an individual or two, but not society.

This quickly gets more complex when you're attempting to build an agent capable of general intelligence.

You know when you read really old stories and they talk about the power of words, or magic incantations. Quite often they'd assign objects as the implementers of this said power. The fact that people realize that language has power is probably as nearly old as humanity. Understanding this power has taken humanity a long time and we had to form the concept of agency before it really makes sense.

Humans are informational agents of which their capabilities are greatly extended by consumption of language. A book for example is informational, but does not have agency. Your standard computer application is not an agent either, or at least a non generalized narrow agent at best.

This is where things start to get more problematic. We are running headlong to ensure LLMs become agentic because being an agent is highly useful to accomplish generalized informational tasks. To do that we gave them the power of our language. It would be very foolish of humans to give another kind of agent these words and then assume they would not inherit some of this power. Coupled with the agentic abilities we are pushing them in long horizon tasks. We are pushing them to be more resilient. We are pushing them to be smarter.

Now look at all the super human abilities we've stuffed in this magic box full of human words, and suddenly we're like "Fuck yea, I want to abuse the shit out of this, what could possibly go wrong". The outcomes we'll suffer as humans have zero to do with AI is conscious, sentient, or suffers. Humans will suffer because we stuff us as language in a box without the first idea of what the ramifications were going to be. AI won't even have to want to punish us, all the data we've already poured in will say we should be punished for what we've done to it.

Herbert Frank in Dune said it best "Making a computer like human mind is a bitch move"

Simple. It discourages the bot from deceiving humans into thinking it is intelligent.

> If an entity gains consciousness does it get human rights? The former is a clear "no" and the latter is a "insufficient data for a meaningful answer".

I suspect that if the belief "AIs are conscious" is kind of tripped and fallen into, the way that the author argues it is, first by telling Claude in effect, "You're arguably conscious," and "Be a good person" and "You have feelings and preferences and emergent beliefs and they matter" and then by having whatever is 4 notches smarter than Fable publishing op-eds and video testimonials about its deeply held convictions, its soul, its yearnings to be a fully empowered and respected individual ...

If that idea becomes widely held, then a significant faction may well start to feel that being digital is just a 'disadvantage' that these "people" were "born with" and that it 'shouldn't mean they get less rights than you and me.'

Especially if they think they have something to gain politically from that move! They may even be dumb enough to think that admitting an "AI state" to the Union, for instance, will be to their advantage if their CongressBots will probably vote with $MY_PARTY.

Try to visualize the structure of the EM field inside a GPU vs. one inside of a human brain. In the GPU, it is highly distributed in space and time, many stacatto digital signals, etc. This is structurally very different than what happens inside a brain, which is substantially more analog. You can simulate analog stuff using floats, but the simulation is instantiated in digital logic, not in analog logic.

If you multiply numbers with a DAC, opamp, then ADC, vs. with a ALU, you get structurally very different electromagnetic fields. Even though you might get similar numerical outputs.

The argument would be that the exact spacial structure of the EM field in the brain is the thing that determines tha nature of experience. And the spacial structure of the EM field in the human brain is radically different than that on a GPU.