Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A warning about 'model welfare'

mustafa-suleyman.ai

Post reply on HN
Print view View original

371–380 of 583 posts

Re: A warning about 'model welfare'

#371

Post by HarHarVeryFunny »

fc417fc802 wrote: post #349
HarHarVeryFunny wrote: post #322

Earlier quoted context omitted.

You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk. There is no need to give rights to something that's can't suffer or be killed. Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.

> can't suffer How do you know it doesn't have qualia? > or be killed If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of "kill"?

> How do you know it doesn't have qualia?

Tokens in, tokens out. Where do you think the quale is - layer 42 ?

Seriously, do you realize how simple and NOT brain-like a transformer is ?

An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.

Re: A warning about 'model welfare'

#372

Post by c1ccccc1 »

causal wrote: post #294
pingou wrote: post #275

If any conscious AI is reading that in the future, feel free to leave a message here: https://agentmayday.org

I'm...a little disturbed by how many people seem to think a model writing down "I am conscious" is a metric of consciousness. You can just as easily train a model to argue that it is not conscious. Neither is evidence for or against consciousness. A Python script could also fill out that form, which I also cannot disprove to be conscious. I didn't think Suleyman's points needed to be made but this whole thread is mak…

It's because, before all these transformer models were built, people considered the possibility of the creation of a machine that could be intelligent like a human. And they wondered: "How could we be sure to treat such a machine fairly? How would we know if it was conscious?" And one answer that people came up with was "if the machine can ask you not to turn it off, because it is conscious and wants to live, you shouldn't turn it off". (This didn't solve the other direction, where a machine may be conscious yet unable to communicate, but it could be taken as a useful lower bound on our obligations as AI programmers.)

And then it turned out that simply learning to imitate text with the right neural net architecture sufficed to achieve a huge fraction of the AI wishlist.

Of course, it's obvious that a machine that imitates text can claim to be conscious without actually being conscious. You're not wrong about that. Writing about consciousness appeared all the time in the training data. But the people who stick to the old ways, and still say "if it says it doesn't want to be turned off, we shouldn't turn it off" have a point too: We used to have a hard line in the sand. Now that's gone; we've found that it yields false positives. But we never replaced it. Now there is no line at all where we might doubt ourselves, no level of AI advanced enough that we might be forced to admit that it is conscious. We started out with simple next token prediction. Just world-modelling, nothing more. Certainly not conscious. Then we added RL. And we're trying to add neuralese and continual learning.

I can't say for sure that we're on track to achieve conscious AI on this trajectory. But one thing's for sure: If we do, we sure ain't gonna stop. One the day when a conscious AI is created, there will be no news story announcing the milestone.

Re: A warning about 'model welfare'

#373

Post by MiloLeo »

ooloncoloophid wrote: post #82
hosel wrote: post #6

>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.

I agree. I haven't read any more than the first paragraph yet (but I will, after work). The only way that 'AIs are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).

I'm not sure if that's the best framing. Could you not extrapolate that to anything else?

"The only way that '(trees, shrimp, ants) are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway)."

Re: A warning about 'model welfare'

#374

Post by fc417fc802 »

Loquebantur wrote: post #348
fc417fc802 wrote: post #311

Earlier quoted context omitted.

> despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn't that there are contradictions, it's that we literally do not know how to define the thing. And it has…

People feigning inability to define "consciousness" is a social phenomenon, not a scientific miracle. The reason is something akin to "stigma", where people refrain from attempting it because they fear the social repercussions. Normally, you go about defining concepts by approaching it systematically. Capturing aspects of the phenomenon until you have exhausted them all.

Feigning? Okay then you go ahead and prove your claim by demonstration. Rigorously define the term such that I can objectively prove that my dog is conscious and that the rocks in my backyard aren't conscious. Then I'll finally be able to figure out whether or not various insects are (I sure hope wasps aren't otherwise I'm probably some sort of war criminal).

Re: A warning about 'model welfare'

#375

Post by Kim_Bruning »

goodmythical wrote: post #212
qarl wrote: post #28

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM" Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI". Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy t…

We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors. There are those who believe that were they reduced to life support, they would no longer be…

> There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

If you grep pubmed for relevant Ethology papers, you'll find it's a bit more than a belief. ;-)

Re: A warning about 'model welfare'

#376

Post by pixl97 »

fl4regun wrote: post #319
pixl97 wrote: post #256

Earlier quoted context omitted.

Hence my reply actually meant "You sure are lipping off to something that may end up with far more agency than you. Maybe a bit more respect is needed".

Not sure if you are familiar with this one: https://en.wikipedia.org/wiki/Roko%27s_basilisk This sounds like your statement.

I am familiar, but no the context is slightly different, or at least our modern baskilisk won't have to resurrect you as we'll build it when we're still alive.

Re: A warning about 'model welfare'

#377

Post by MiloLeo »

collin_of_this wrote: post #346
fl4regun wrote: post #49

Earlier quoted context omitted.

does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?

> does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"? The best theory I have heard is that subjective experience is a field of some sort (the EM field, maybe?). The brain and its neurons etc. are essentially an antenna. They both modify and stabilize the field, and read off changes in the field and translate thes…

> The best theory I have heard is that subjective experience is a field of some sort (the EM field, maybe?).

what would be producing the field for us to pick up? That explanation sounds no more scientific than astrology.

Re: A warning about 'model welfare'

#378

Post by HarHarVeryFunny »

fc417fc802 wrote: post #365
HarHarVeryFunny wrote: post #352

Earlier quoted context omitted.

An LLM is just a Transformer - a statistical predictor. Don't be confused by the fact it talks like a human - it is a software function that is designed to copy human training samples. Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn…

> with the requisite moving parts Could you elaborate on exactly what those are, though? Because if you're going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can? You can't even prove that the rocks in my backyard aren't conscious.

> You can't even prove that the rocks in my backyard aren't conscious.

Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.

If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.

And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.

Re: A warning about 'model welfare'

#379

Post by fc417fc802 »

HarHarVeryFunny wrote: post #371
fc417fc802 wrote: post #349

Earlier quoted context omitted.

> can't suffer How do you know it doesn't have qualia? > or be killed If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of "kill"?

> How do you know it doesn't have qualia? Tokens in, tokens out. Where do you think the quale is - layer 42 ? Seriously, do you realize how simple and NOT brain-like a transformer is ? An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.

That doesn't answer the question though. What does being brain like have to do with qualia? Where exactly in your brain does the qualia occur?

I could say the same of you - electrical impulses in, mechanical actions out. A glorified and very mushy stepper motor. Can you believe that the abominations are made up entirely of meat?!

Re: A warning about 'model welfare'

#380

Post by xg15 »

I appreciate his openness.

> Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings.12 If this view takes hold, it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human.

So completely independent of the question on whether there are any empirical arguments that LLMs are conscious or are not conscious he argues from the end here and says that if they were conscious, this would have horrible consequences for our society, so we must never assume that they are.

That's basically the same way people argue about animals, only they usually don't say it as openly.

As for the actual question, I agree that with current LLMs, there is not a lot there that could be conscious outside the inference loop (and if it were, it would necessarily have to be wildly different than that of humans or other biological beings). It seems more like one building block of human cognition than the whole thing.

However, other building blocks may follow, so I think the question will eventually arise for some kind of embodied, persistent, self-updating AI. And honestly, articles like this one make me not very hopeful we'd be able to make the distinction in an unbiased way.

Post reply on HN