A warning about 'model welfare' • Hacker News Board

Re: A warning about 'model welfare'

#621

Post by myrmidon » Thu, Sep 17, 2026, 8:56 AM UTC

myaccountonhn wrote: post #612

myrmidon wrote: post #143

I have a very simple benchmark for arguments on AI ethics: Substitute black people/women/animals as subject (instead of AI). Does that make you sound like a well-known moustache wearer? Then your argument is bad and needs work. This clearly falls into that category.

My issue is that I've seen some bad takes in rationalist circles that believe in functionalism, and consciousness as an emergent property of a sufficiently complex computation. Using that as their foundation, they've then argued that most animals are not conscious (or less conscious), but LLMs are.

That is wild.

I strongly believe that any definition of consciousness that excludes all animals is unusable BS (but people love to argue about consciousness without defining it at all, anyway).

Re: A warning about 'model welfare'

#622

Post by vidarh » Thu, Sep 17, 2026, 9:00 AM UTC

chrisjj wrote: post #513

vidarh wrote: post #124

Earlier quoted context omitted.

The model does not. The system it runs in can. My agents have persistent memory. It is not at all clear whether or the distinction that the long-term memory is external from the model matters.

> My agents have persistent memory. So your disc drive likewise experiences emotions?

Does your disc drive have the complexity of an LLM that shows behaviour consistent with having emotions?

I'd consider that a prerequisite, but not sufficient, for that question to make sense.

Re: A warning about 'model welfare'

#623

Post by vidarh » Thu, Sep 17, 2026, 9:02 AM UTC

willy_k wrote: post #564

ooloncoloophid wrote: post #91

Earlier quoted context omitted.

What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?

I do. God. I am partial to and bullish on the brain being a quantum-classical computer. There has been interesting research exploring that recently [0, 1, 2, 3]. That (quantum mechanics) seems to be escape-hatch, so to speak, from the otherwise deterministic, mechanical procession of the universe. [0]** https://www.researchgate.net/publication/15280350_Quantum_op... [1] https://pubs.acs.org/jpcbfk/article-pdf/128/17/…

For it to be an "escape hatch", there'd need to be a measurably different distribution of outputs than what you'd otherwise expect, that translates to functions that are not Turing computable. There is zero evidence for that.

Re: A warning about 'model welfare'

#624

Post by suddenlybananas » Thu, Sep 17, 2026, 9:04 AM UTC

myrmidon wrote: post #620

suddenlybananas wrote: post #470

Earlier quoted context omitted.

If you change the most important detail of an argument, it may no longer be sound! Wow!

Thats the thing though: If the core part of your ethics argument is just using dehumanizing language to justify why it's ok to maintain the status quo then your argument is bad. Comparing specifically to pro-slavery perspectives makes that easy to see because those arguments from slave owners did basically the exact same thing (and were completely wrong/misguided in hindsight).

>de"human"izing

Slave owners did it to people, not software. This comparison is just begging the question, as whether or not software is of moral weight comparable to humans is precisely what is at issue.

Re: A warning about 'model welfare'

#625

Post by tim333 » Thu, Sep 17, 2026, 9:10 AM UTC

snickerbockers wrote: post #492

tim333 wrote: post #487

>AIs are not conscious... If humanity is to flourish in the 21st century, that is how they must remain. I don't think that's true - researching consciousness by trying to build conscious AI seems natural step forward in understanding ourselves. I don't see how it'd stop flourishing particularly.

So what you're saying is that it's conscious as long as the guy who made it wants it to be? I know consciousness is notoriously difficult to define or prove to anybody other than yourself but thay doesn't mean anything which a third party believes to be conscious is actually conscious.

Not really. I mean there's probably scientific study of how it works in brains and you can try copying the kind of design in software and see how it compares.

Re: A warning about 'model welfare'

#626

Post by myrmidon » Thu, Sep 17, 2026, 9:31 AM UTC

suddenlybananas wrote: post #624

myrmidon wrote: post #620

Earlier quoted context omitted.

Thats the thing though: If the core part of your ethics argument is just using dehumanizing language to justify why it's ok to maintain the status quo then your argument is bad. Comparing specifically to pro-slavery perspectives makes that easy to see because those arguments from slave owners did basically the exact same thing (and were completely wrong/misguided in hindsight).

>de"human"izing Slave owners did it to people , not software. This comparison is just begging the question, as whether or not software is of moral weight comparable to humans is precisely what is at issue.

Slave owners certainly convinced themselves that those were not "real people".

This article is exactly the same; all it talks about is how inconvenient and dangerous it could be for AI to be granted rights, while the core assumptions ("consciousness is very likely biological") rest on the shakiest of grounds.

Re: A warning about 'model welfare'

#627

Post by myrmidon » Thu, Sep 17, 2026, 9:36 AM UTC

voidnullvalue wrote: post #426

myrmidon wrote: post #264

Earlier quoted context omitted.

No, my position is that "black people/women/animals should do repetitive work so I don't have to" does not qualify because it doesn't make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it's basically the status quo. Compare: " are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by " "granting rights and imb…

What properties do LLMs possess that make human-animal moral status analogies appropriate?

Ability to think/reason.

Re: A warning about 'model welfare'

#628

Post by suddenlybananas » Thu, Sep 17, 2026, 9:39 AM UTC

myrmidon wrote: post #626

suddenlybananas wrote: post #624

Earlier quoted context omitted.

>de"human"izing Slave owners did it to people , not software. This comparison is just begging the question, as whether or not software is of moral weight comparable to humans is precisely what is at issue.

Slave owners certainly convinced themselves that those were not "real people". This article is exactly the same; all it talks about is how inconvenient and dangerous it could be for AI to be granted rights, while the core assumptions ("consciousness is very likely biological") rest on the shakiest of grounds.

That's not really true. Slave owners were perfectly aware that slaves were humans.

Re: A warning about 'model welfare'

#629

Post by myrmidon » Thu, Sep 17, 2026, 9:55 AM UTC

suddenlybananas wrote: post #628

myrmidon wrote: post #626

Earlier quoted context omitted.

Slave owners certainly convinced themselves that those were not "real people". This article is exactly the same; all it talks about is how inconvenient and dangerous it could be for AI to be granted rights, while the core assumptions ("consciousness is very likely biological") rest on the shakiest of grounds.

That's not really true. Slave owners were perfectly aware that slaves were humans.

Black people being supposedly "inferior sub-humans" was one of the primary justifications for their treatment.