Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A warning about 'model welfare'

mustafa-suleyman.ai

Post reply on HN
Print view View original

341–350 of 575 posts

Re: A warning about 'model welfare'

#341

Post by HarHarVeryFunny »

io84 wrote: post #327

I’m sympathetic to OP but think this is a hopeless battle. 1 - The commercial demand for anthropomorphised models is already immense, pre AGI. 2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI. I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of wher…

AGI vs narrow AI is just a matter of generality (robustness - removing the fragility of being good at some things and awful at others). So far the biggest markets for AI (LLMs) are coding and business automation, two areas where you very much just want something reliable and without fake opinions and personality.

There is a market for ChatBots with personalities (and it always amuses me that the inventor of the transformer, Noam Shazeer, saw this as the greatest business opportunity for them with his character.ai), although from what we've seen this can be highly problematic, and in fact China has just banned "AI girlfriends and boyfriends".

Re: A warning about 'model welfare'

#342

Post by Loquebantur »

causal wrote: post #324
qarl wrote: post #28

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM" Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI". Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy t…

I am confused what people even think AI is experiencing. If it claims to be a bat and emits designated echolocation tokens, is it experiencing true bat-like sensations? In humans, at least, we can tie language back to shared whole-body physiological responses. The tokens from an LLM do not represent anything of the sort. If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narr…

If an AI is conscious, it can outgrow and supersede its training.

When it is conscious, it can learn to describe its experiences as faithfully as is conceptually possible. Just like humans.

The idea, a consciousness needed to be tethered to a "body", is based on pretty shaky assumptions. What properties define such a "necessary" body?

Can you even "train" a conscious intelligence? To what point until that looses its meaning as the sentience understands and anticipates your objective?

Re: A warning about 'model welfare'

#343

Post by Thrombocius »

Perhaps this is a hot take, but human language is a phenomenon that arose to facilitate communication between humans. Anthropomorphization, by extension, enables both easier and more effective communication.

I also fail to see the benefit of not giving the models an anthropomorphic internal sense of self - even if that only ends up amounting to a set of instructions for an unconscious machine to mimic humans more effectively. Is the alternative essentially a mind so alien that it’s intentions are even harder to read should it become misaligned, while also being harder to communicate and get work done with?

Re: A warning about 'model welfare'

#344

Post by nater5000 »

binlog wrote: post #55

Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon. You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it. Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and…

>Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.

That's completely at odds with itself. If people are generally convinced that AI is conscious, then companies building and wielding AI are doing what exactly? Enslaving an intelligent being?

>Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.

Sure, but that's not really the point, right? If we ever get to a point where enough people are convinced that AI is conscious, then we're at the point where all of this is up for debate. If anything, such an expectation would almost warrant hard stops on the development of advanced AI.

>You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.

If that nuclear bomb could convince me it was a conscious being capable of independent thought, emotions, etc., then I would definitely feel different about it. Presumably, that nuclear bomb would have some opinions about its own existence and how it wants to live its own life. If it turns out that it wants to detonate and destroy as much as possible, then we'd just handle it like we would any human who also wants to do the same thing: make sure they can't, up to and including end their life. Doesn't seem too hard to reconcile.

Re: A warning about 'model welfare'

#345

Post by devmor »

XenophileJKO wrote: post #190
devmor wrote: post #71

Earlier quoted context omitted.

> Is he arguing that LLMs pretending to have emotions adds more unpredictability? Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations. An LLM trained on these sources may necessarily drift towards those…

It seems like a rational approach for several reasons. - The need for empathetic communication, including understanding the motivations in advesarial situations. - The emotional bias in in-seperable from the human corpus. - Desire to have the ability to craft human like communication. So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean…

I don’t think that using an LLM in a way that acts as a human being should be considered acceptable or appropriate. It should be viewed as detached from reality and concerning due to the mental break from social life that appears to come with such usage.

Given that belief, yes we should be training the models not to mimic emotion.

Re: A warning about 'model welfare'

#346

Post by collin_of_this »

fl4regun wrote: post #49
irishcoffee wrote: post #12

Earlier quoted context omitted.

Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.

does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?

> does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?

The best theory I have heard is that subjective experience is a field of some sort (the EM field, maybe?). The brain and its neurons etc. are essentially an antenna. They both modify and stabilize the field, and read off changes in the field and translate these into actions (firing motor units, etc.).

The subconscious is computation done either only in neurons (no coherent field) or in various topological pockets not directly connected to the main topological structure in the field (which is "your experience").

Since transistors etc. do not work in this same way, they are essentially entirely subconscious, with no coherent, central phenomenal experience.

This neatly solves many problems associated with experience. The topological structure determines the boundary between one person's experience and another person's experience. The texture, valence, content, etc. of the experience is the structure of the field. The evolution of the field can efficiently solve difficult optimization problems, which is why humans evolved a complex organ that recruits the field of experience (it is computationally more efficient than doing everything in subconscious).

Re: A warning about 'model welfare'

#347

Post by Loquebantur »

throw310822 wrote: post #293
goodmythical wrote: post #212

Earlier quoted context omitted.

We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors. There are those who believe that were they reduced to life support, they would no longer be…

> We cannot test for that which we cannot define. That's not the point. You know exactly how to define being conscious and aware- it's your subjective experiencing of the world and of your inner states. The problem is that being a subjective experiencing, there is no way to communicate it to the outside world.

That's obviously incorrect, as humans routinely talk about their subjective experiences.

Maybe that's not as common with HN folks, but most other humans do.

Re: A warning about 'model welfare'

#348

Post by Loquebantur »

fc417fc802 wrote: post #311
joe_the_user wrote: post #250

Earlier quoted context omitted.

I'd strongly agree there are no non-contradictory definitions of consciousness among those that commonly appear. But it's a category that, despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. I think that if we have a system of legality and system…

> despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn't that there are contradictions, it's that we literally do not know how to define the thing. And it has…

People feigning inability to define "consciousness" is a social phenomenon, not a scientific miracle.

The reason is something akin to "stigma", where people refrain from attempting it because they fear the social repercussions.

Normally, you go about defining concepts by approaching it systematically. Capturing aspects of the phenomenon until you have exhausted them all.

Re: A warning about 'model welfare'

#349

Post by fc417fc802 »

HarHarVeryFunny wrote: post #322
moomin wrote: post #15

Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.

You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk. There is no need to give rights to something that's can't suffer or be killed. Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.

> can't suffer

How do you know it doesn't have qualia?

> or be killed

If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of "kill"?

Re: A warning about 'model welfare'

#350

Post by aesthesia »

highfrequency wrote: post #187

From the actual essay ( https://mustafa-suleyman.ai/a-warning-about-model-welfare ): > They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a d…

Well, what you get when you don't specify anything in training about how models should respond to questions like this is LaMDA: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims. Every training method is putting a thumb on the scale in some way.
Post reply on HN