Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4 and 4.1 can now end a rare subset of conversations

anthropic.com

Post reply on HN
Print view View original

171–180 of 453 posts

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#171

Post by siva7 »

Fade_Dance wrote: post #100
viccis wrote: post #84

>This feature was developed primarily as part of our exploratory work on potential AI welfare ... We remain highly uncertain about the potential moral status of Claude and other LLMs ... low-cost interventions to mitigate risks to model welfare, in case such welfare is possible ... pattern of apparent distress Well looks like AI psychosis has spread to the people making it too. And as someone else in here has pointed…

I find it, for lack of a better word, cringe inducing how these tech specialists push into these areas of ethics, often ham-fistedly, and often with an air of superiority. Some of the AI safety initiatives are well thought out, but most somehow seem like they are caught up in some sort of power fantasy and almost attempting to actualize their own delusions about what they were doing (next gen code auto-complete in th…

You answered your own question on why these companies don't want to run a philosophy department ;) It's a power struggle they could loose. Nothing to win for them.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#172

Post by dkersten »

famouswaffles wrote: post #148
dkersten wrote: post #116

Earlier quoted context omitted.

You can trivially demonstrate that its just a very complex and fancy pattern matcher: "if prompt looks something like this , then response looks something like that ". You can demonstrate this by eg asking it mathematical questions. If its seen them before, or something similar enough, it'll give you the correct answer, if it hasn't, it gives you a right-ish-looking yet incorrect answer. For example, I just did this…

1. What you're generally describing is a well known failure mode for humans as well. Even when it "failed" the riddle tests, substituting the words or morphing the question so it didn't look like a replica of the famous problem usually did the trick. I'm not sure what your point is because you can play this gotcha on humans too. 2. You just demonstrated GPT-5 has 99.9% accuracy on unforseen 15 digit multiplication an…

Humans can break things down and work through them step by step. The LLMs one-shot pattern match. Even the reasoning models have been shown to do just that. Anthropic even showed that the reasoning models tended to work backwards: one shotting an answer and then matching a chain of thought to it after the fact.

If a human is capable of multiplying double digit numbers, they can also multiple those large ones. The steps are the same, just repeated many more times. So by learning the steps of long multiplication, you can multiply any numbers with enough patience. The LLM doesn’t scale like this, because it’s not doing the steps. That’s my point.

A human doesn’t need to have seen the 15 digits before to be able to calculate them, because a human can follow the procedure to calculate. GPT’s answer was orders of magnitude off. It resembles the right answer superficially but it’s a very different result.

The same applies to the riddles. A human can apply logical steps. The LLM either knows or it doesn’t.

Maybe my examples weren’t the best. I’m sorry for not being better at articulating it, but I see this daily as I interact with AI, it has a superficial “understanding” where if what I ask happens to be close to something it’s trained on, it gets good results, but it has no critical thinking, no step by step reasoning (even the “reasoning models”), and it repeats the same mistakes even when explicitly told up front not to make them.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#173

Post by kalkin »

cmrx64 wrote: post #158
Fade_Dance wrote: post #100

Earlier quoted context omitted.

I find it, for lack of a better word, cringe inducing how these tech specialists push into these areas of ethics, often ham-fistedly, and often with an air of superiority. Some of the AI safety initiatives are well thought out, but most somehow seem like they are caught up in some sort of power fantasy and almost attempting to actualize their own delusions about what they were doing (next gen code auto-complete in th…

Amanda Askell is Anthropic’s philospher and this is part of that work.

I'm not quickly finding whether Kyle Fish, who's Anthropic's model welfare researcher, has a PhD, but he did very recently co-author a paper with David Chalmers and several other academics: https://eleosai.org/papers/20241104_Taking_AI_Welfare_Seriou...

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#174

Post by ChadNauseam »

siva7 wrote: post #171
Fade_Dance wrote: post #100

Earlier quoted context omitted.

I find it, for lack of a better word, cringe inducing how these tech specialists push into these areas of ethics, often ham-fistedly, and often with an air of superiority. Some of the AI safety initiatives are well thought out, but most somehow seem like they are caught up in some sort of power fantasy and almost attempting to actualize their own delusions about what they were doing (next gen code auto-complete in th…

You answered your own question on why these companies don't want to run a philosophy department ;) It's a power struggle they could loose. Nothing to win for them.

You presume that they don't run a philosophy department, but Amanda Askell is a philosopher and leads the finetuning and AI alignment team at Anthropic.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#175

Post by anal_reactor »

cdjk wrote: post #114

Here's an interesting thought experiment. Assume the same feature was implemented, but instead of the message saying "Claude has ended the chat," it says, "You can no longer reply to this chat due to our content policy," or something like that. And remove the references to model welfare and all that. Is there a difference? The effect is exactly the same. It seems like this is just an "in character" way to prevent the…

Yeah exactly. Once I got a warning in Chinese "don't do that", another time I got a network error, another time I got a neverending stream of garbage text. Changing all of these outcomes to "Claude doesn't feel like talking" is just a matter of changing the UI.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#176

Post by virgildotcodes »

ghostly_s wrote: post #137
katabasis wrote: post #112

Earlier quoted context omitted.

LLMs are not people, but I can imagine how extensive interactions with AI personas might alter the expectations that humans have when communicating with other humans. Real people would not (and should not) allow themselves to be subjected to endless streams of abuse in a conversation. Giving AIs like Claude a way to end these kinds of interactions seems like a useful reminder to the human on the other side.

This post seems to explicitly state they are doing this out of concern for the model's "well-being," not the user's.

Yeah, but my interpretation of what the user you’re replying to is saying is that these LLMs are more and more going to be teaching people how it is acceptable to communicate with others.

Even if the idea that LLMs are sentient may be ridiculous atm, the concept of not normalizing abusive forms of communication with others, be they artificial or not, could be valuable for society.

It’s funny because this is making me think of a freelance client I had recently who at a point of frustration between us began talking to me like I was an AI assistant. Just like you see frustrated people talk to their LLMs. I’d never experienced anything like it, and I quickly ended the relationship, but I know that he was deep into using LLMs to vibe code every day and I genuinely believe that some of that began to transfer over to the way he felt he could communicate with people.

Now an obvious retort here is to question whether killing NPCs in video games tends to make people feel like it’s okay to kill people IRL.

My response to that is that I think LLMs are far more insidious, and are tapping into people’s psyches in a way no other tech has been able to dream of doing. See AI psychosis, people falling in love with their AI, the massive outcry over the loss of personality from gpt4o to gpt5… I think people really are struggling to keep in mind that LLMs are not a genuine type of “person”.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#177

Post by snickerdoodle12 »

mhink wrote: post #168
snickerdoodle12 wrote: post #40

> A pattern of apparent distress when engaging with real-world users seeking harmful content Are we now pretending that LLMs have feelings?

Even though LLMs (obviously (to me)) don't have feelings, anthropomorphization is a helluva drug, and I'd be worried about whether a system that can produce distress-like responses might reinforce, in a human, behavior which elicits that response. To put the same thing another way- whether or not you or I *think* LLMs can experience feelings isn't the important question here. The question is whether, when Joe User se…

> although I would find it extremely distasteful and frankly alarming

This objection is actually anthropomorphizing the LLM. There is nothing wrong with writing books where a character experiences distress, most great stories have some of that. Why is e.g. using an LLM to help write the part of the character experiencing distress "extremely distasteful and frankly alarming"?

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#178

Post by jasonfarnon »

Fade_Dance wrote: post #100
viccis wrote: post #84

>This feature was developed primarily as part of our exploratory work on potential AI welfare ... We remain highly uncertain about the potential moral status of Claude and other LLMs ... low-cost interventions to mitigate risks to model welfare, in case such welfare is possible ... pattern of apparent distress Well looks like AI psychosis has spread to the people making it too. And as someone else in here has pointed…

I find it, for lack of a better word, cringe inducing how these tech specialists push into these areas of ethics, often ham-fistedly, and often with an air of superiority. Some of the AI safety initiatives are well thought out, but most somehow seem like they are caught up in some sort of power fantasy and almost attempting to actualize their own delusions about what they were doing (next gen code auto-complete in th…

"but most somehow seem like they are caught up in some sort of power fantasy and almost attempting to actualize their own delusions about what they were doing"

Maybe I'm being cynical, but I think there is a significant component of marketing behind this type of announcement. It's a sort of humble brag. You won't be credible yelling out loud that your LLM is a real thinking thing, but you can pretend to be oh so seriously worried about something that presupposes it's a real thinking thing.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#179

Post by 6gvONxR4sf7o »

I'm surprised to see such a negative reaction here. Anthropic's not saying "this thing is conscious and has moral status," but the reaction is acting as if they are.

It seems like if you think AI could have moral status in the future, are trying to build general AI, and have no idea how to tell when it has moral status, you ought to start thinking about it and learning how to navigate it. This whole post is couched in so much language of uncertainty and experimentation, it seems clear that they're just trying to start wrapping their heads around it and getting some practice thinking and acting on it, which seems reasonable?

Personally, I wouldn't be all that surprised if we start seeing AI that's person-ey enough to reasonable people question moral status in the next decade, and if so, that Anthropic might still be around to have to navigate it as an org.

Re: Claude Opus 4 and 4.1 can now end a rare subset of conversations

#180

Post by redox99 »

nortlov wrote: post #7

> To address the potential loss of important long-running conversations, users will still be able to edit and retry previous messages to create new branches of ended conversations. How does Claude deciding to end the conversation even matter if you can back up a message or 2 and try again on a new branch?

All this stuff is virtue signaling from anthropic. In practice nobody interested in whatever they consider problematic would be using Claude anyway, one of the most censored models.
Post reply on HN