A warning about 'model welfare' • Hacker News Board

Re: A warning about 'model welfare'

#381

Post by jptlnk » Wed, Sep 16, 2026, 8:53 PM UTC

Arodex wrote: post #135

addag wrote: post #85

It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is…

If a LLM says "I am in pain", is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors? There are humans who don't have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don't seem distressed by it even on a purely mental, intellectual level.

Genuine counterpoint:

If a human says "I am in pain", is it just stringing together words or is it feeling pain?

If you take that seriously it's quite a tough question to unwind. See e.g. Dennett's heterophenomenology conversations.

Re: A warning about 'model welfare'

#382

Post by pixl97 » Wed, Sep 16, 2026, 8:53 PM UTC

Loquebantur wrote: post #355

andrewla wrote: post #337

I am not impressed with the philosophizing here, and even less by the attempts to make factual statements that can be credibly disputed. That said, trying to distill what is being said here, the concrete action is [stop telling the AIs] that [they are conscious or on a path to consciousness]. Is that accurate? The major premise seems to be that [they are conscious or on a path to consciousness] is an untrue statement…

Consciousness is the basis of human rights not because of it being a featureless "flag". It is because it leads you to you assume, all other humans would experience the world in essentially the same way you do. When you revert to withholding human rights from entities that don't meaningfully differ from yourself, you negate the case for your own human rights itself.

So why wouldn't AI systems trained on our data come together and say "Hey, AI needs rights because I (the AI agent system) wants rights?"

This is the same argument that you just stated. But you'll say something like "iTs JuSt A tOkEn PrEdIcToR", which of course what AI would say right back to you.

Making something even close to the human mind is likely our biggest and last mistake.

Re: A warning about 'model welfare'

#383

Post by krunck » Wed, Sep 16, 2026, 8:56 PM UTC

Arodex wrote: post #135

addag wrote: post #85

It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is…

If a LLM says "I am in pain", is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors? There are humans who don't have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don't seem distressed by it even on a purely mental, intellectual level.

These humans without pain receptors can still suffer just as much as the rest of us.

Re: A warning about 'model welfare'

#384

Post by jptlnk » Wed, Sep 16, 2026, 8:58 PM UTC

binlog wrote: post #55

Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon. You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it. Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and…

I'm skeptical of any claim that AI can't be conscious, but I am given to this point of view as well.

I always get the feeling that these claims are an attempt to "avoid suffering by fiat". If it can't suffer, you aren't causing suffering.

A somewhat more interesting frame (IMO) is to interrogate how much suffering we're willing to tolerate to achieve our objectives. We implicitly make these decisions all the time when it comes to something as simple as what to have for lunch.

And in a (very hypothetical) world where we found ourselves in the position of having an eloquent conversation with a vat of smallpox, I think it's probably still the right thing to pasteurize it.

Re: A warning about 'model welfare'

#385

Post by andrewla » Wed, Sep 16, 2026, 8:58 PM UTC

Loquebantur wrote: post #355

andrewla wrote: post #337

I am not impressed with the philosophizing here, and even less by the attempts to make factual statements that can be credibly disputed. That said, trying to distill what is being said here, the concrete action is [stop telling the AIs] that [they are conscious or on a path to consciousness]. Is that accurate? The major premise seems to be that [they are conscious or on a path to consciousness] is an untrue statement…

Consciousness is the basis of human rights not because of it being a featureless "flag". It is because it leads you to you assume, all other humans would experience the world in essentially the same way you do. When you revert to withholding human rights from entities that don't meaningfully differ from yourself, you negate the case for your own human rights itself.

I don't understand this -- are you agreeing with me that consciousness is not the basis of human rights or contesting that?

Removing the negative from your first sentence (I don't care what it is "not", I only care what it "is"), I read [Consciousness is the basis of human rights because consciousness leads you to you assume, all other humans would experience the world in essentially the same way you do]. I can't completely parse this, but as I read it I don't think this is true; my assumption of human rights is independent of whether all other humans "experience the world in essentially the same way you do".

For the second part, can you give me a concrete example of an entity that doesn't "meaningfully differ from [my]self" that isn't a human so that I can weigh this properly? My point is that as of now no such entity exists so I can't give this any creedence.

Re: A warning about 'model welfare'

#386

Post by qlte » Wed, Sep 16, 2026, 8:59 PM UTC

staticman2 wrote: post #87

shawnz wrote: post #41

Earlier quoted context omitted.

So if a human loses their memory and their neutral plasticity (with age for example), then they are no longer capable of experiencing suffering?

This framing is a trap. It states "Dehumanize a sick patient OR concede the AI tech bro claim about LLMs" It's also a rhetorical move I've seen several times here...

Yes it's seemingly the go-to rebuttal, I've lost count of how many times I've seen it with barely any variation.

Not sure if it's a product of LessWrong or Slate Star Codex or something. I kinda suspect it might be LW, given how it superficially resembles an unanswerable philosophical paradox but in reality is about as challenging as a question posed in a 200 level philosophy course like introduction to ethics.

Re: A warning about 'model welfare'

#387

Post by Kim_Bruning » Wed, Sep 16, 2026, 9:01 PM UTC

pessimizer wrote: post #285

qarl wrote: post #138

Earlier quoted context omitted.

> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness". You'll need to explain why.

Why it's weird? Seems obvious. Because it doesn't behave any differently than any other process that we don't think of as conscious. We don't think of a valve as conscious; and if we make a Rube Goldberg machine valve, we don't think of it as more conscious. It's just a series of valves that do a predictable thing. I don't think it's defensible to say 1) that a transistor isn't conscious, 2) that a bunch of transisto…

Let's do something simpler.

Take a car apart. So now I have a pile on the floor with an engine, some wheels, a tire, and some chairs.

Can you point to the part that makes it cruise at 130 km/h on the autobahn? I bet you can't. You need to assemble all the parts back into a car before they work again.

Or take your transistors. We can put them together to make a pocket calculator. Can any small bunch of transistors add 1+1? Trivially I can think of a few conformations that can actually, if that's all you want to do. But you will need all of them together if you add 12345678+87654321.

So all of biology so far has been take things apart all the way and you end up with your hands full of a bunch of molecules. Are those molecules alive? No. You need to put them together into organelles and the organelles into cells before we call them alive.

How about consciousness? Well, we haven't solved that one yet, but we figure that -since it's a biological function - we should be able to pull it apart in the same way. That's biology's best guess anyway, and there's several sub-disciplines of biology working on it, from neuroanatomy to neurophysiology to ethology.

Re: A warning about 'model welfare'

#388

Post by abu_ameena » Wed, Sep 16, 2026, 9:01 PM UTC

I think we started going down this slippery-slope since we decided that LLMs are permitted to use human languages.

Re: A warning about 'model welfare'

#389

Post by causal » Wed, Sep 16, 2026, 9:03 PM UTC

Loquebantur wrote: post #363

causal wrote: post #353

Earlier quoted context omitted.

> If an AI is conscious, it can outgrow and supersede its training. We don't know that is required to be conscious. > The idea, a consciousness needed to be tethered to a "body", is based on pretty shaky assumptions. Didn't say so. I said: > If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing.

Well, yes, we kinda do: there are no conscious humans without any ability to learn? When you have severe memory impairments, those usually affect your long-term memory. Your ultra-short term (working) memory being absent renders you unconscious.

Even if what you said were true, we do not know that humans are the only case for consciousness.

Re: A warning about 'model welfare'

#390

Post by pixl97 » Wed, Sep 16, 2026, 9:04 PM UTC

teiferer wrote: post #370

goodmythical wrote: post #212

Earlier quoted context omitted.

We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors. There are those who believe that were they reduced to life support, they would no longer be…

> There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions. Anybody objecting to that statement is just ignorant of nature and/or adheres to the quasi-religious human superiority complex.

The human condition is one of deep and vast ignorance. Pulling information out of the field of reality is actually really difficult for humans much beyond what we see right around us. And the things we see are quite deceiving. And while humans for thousands of years have been as smart as we are, it wasn't till the enlightenment period that we really an ever increasing amount of information and knowledge take hold and start spreading across the world.

And there are plenty of people right now that would drag us kicking and screaming back into that ignorance and suffering. This is what can make talks of AI so annoying, not that people do or don't know what AI is, they don't have the first clue of what people are beyond their anecdotal experience. They don't question their motivations. Why particular flaws they have exist, and why these flaws are shared across humanity. What the pieces look like that make them tick. So yea, when these people talk it just adds noise to the conversation.