Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic walks back policy that could have 'sabotaged' researchers using Claude

wired.com

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#11

Post by LoganDark »

chatmasta wrote: post #10
LoganDark wrote: post #8

Earlier quoted context omitted.

I think they are legitimately convinced that this model is so dangerous it could destroy the world and that they genuinely have the responsibility to prevent it from assisting other models to destroy the world. I don't think I agree that I should be forbidden from e.g. patching a binary to work on the latest macOS since the company behind it died and intentionally installed a time-based kill-switch (FUCK ADOBE for po…

The company was founded basically out of the effective altruism movement.

What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#12

Post by chatmasta »

LoganDark wrote: post #11
chatmasta wrote: post #10

Earlier quoted context omitted.

The company was founded basically out of the effective altruism movement.

What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.

It’s logical consideration with “logical” meaning Spock style logic, ie utilitarianism at all costs. Another prominent EA is SBF for example. It’s designed to sound innocuous and many of its cultish promoters may genuinely believe it’s innocuous, but it’s not.

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#13

Post by reasonableklout »

nmfisher wrote: post #5

Call me a cynic, but I don't believe this is a genuine change of heart at all. It feels much more like a panicked response to something that might undermine their IPO. Even if you trust Anthropic today (which I don't), they clearly don't want competition and there's no telling what other shady moves they'll pull in future. The only sustainable way forward is to support open models. I was already on the fence about wh…

I guess I don't understand why it's shady. It seems more like a poorly executed decision to enforce a publicly stated policy (it's been against Anthropic's ToS to use their models on frontier ML research for a while now). After all, people found out about this through their published system card.

It is definitely a bad idea to do this without notifying the user, because users who are incorrectly affected will have no way of providing feedback or getting support. And it is also anticompetitive, but if you truly believe that AI is not a normal technology, it is rational.

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#14

Post by LoganDark »

chatmasta wrote: post #12
LoganDark wrote: post #11

Earlier quoted context omitted.

What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.

It’s logical consideration with “logical” meaning Spock style logic, ie utilitarianism at all costs. Another prominent EA is SBF for example. It’s designed to sound innocuous and many of its cultish promoters may genuinely believe it’s innocuous, but it’s not.

Can you help me understand what costs? Utilitarianism alone would not necessarily be so obsessed with these safeguards -- Anthropic seems to have much more of an obsession with moral good than utilitarianism alone would suggest. I feel utilitarianism alone would likely be more obsessed with advancing the technology, making it generally available, and more generally compensating for attackers advancing at similar rates, than with obsessively trying to avoid being the way they get there. In other words, utilitarianism alone wouldn't explain such the obsessive sense of responsibility and fear of reprehensibility over how their tools are used.

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#15

Post by SilverElfin »

LoganDark wrote: post #8
nmfisher wrote: post #5

Call me a cynic, but I don't believe this is a genuine change of heart at all. It feels much more like a panicked response to something that might undermine their IPO. Even if you trust Anthropic today (which I don't), they clearly don't want competition and there's no telling what other shady moves they'll pull in future. The only sustainable way forward is to support open models. I was already on the fence about wh…

I think they are legitimately convinced that this model is so dangerous it could destroy the world and that they genuinely have the responsibility to prevent it from assisting other models to destroy the world. I don't think I agree that I should be forbidden from e.g. patching a binary to work on the latest macOS since the company behind it died and intentionally installed a time-based kill-switch (FUCK ADOBE for po…

> I think they are legitimately convinced that this model is so dangerous it could destroy the world and that they genuinely have the responsibility to prevent it from assisting other models to destroy the world.

Do they really believe that? Or do they just want to control this technology exclusively with moves like this and with pushing for regulatory capture after complaining about safety all the time? Didn’t Dario say that GPT2 or GPT3 would present a similar destroy the world level of danger?

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#17

Post by nmfisher »

reasonableklout wrote: post #13
nmfisher wrote: post #5

Call me a cynic, but I don't believe this is a genuine change of heart at all. It feels much more like a panicked response to something that might undermine their IPO. Even if you trust Anthropic today (which I don't), they clearly don't want competition and there's no telling what other shady moves they'll pull in future. The only sustainable way forward is to support open models. I was already on the fence about wh…

I guess I don't understand why it's shady. It seems more like a poorly executed decision to enforce a publicly stated policy (it's been against Anthropic's ToS to use their models on frontier ML research for a while now). After all, people found out about this through their published system card. It is definitely a bad idea to do this without notifying the user, because users who are incorrectly affected will have no…

It's shady because they were going to silently poison your outputs.

It's actually worse than it sounds initially, because Fable isn't actually omniscient when it comes to safety classification. Many people (myself included) had refusals or fallback to Opus 4.8 for seemingly compliant/innocuous requests.

Wouldn't you be pissed off if they decided to sabotage your project despite having done nothing wrong?

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#18

Post by EnPissant »

LoganDark wrote: post #11
chatmasta wrote: post #10

Earlier quoted context omitted.

The company was founded basically out of the effective altruism movement.

What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.

It's not real. It's like naming your movement "The Good People". It sprouted from the "Rationalist" community, which is even more self-aggrandizing.

Neither has any hope of doing any good for the world as they don't understand evolutionary pressures. They are set up to reward making members feel smart, not accomplishing anything.

And if they ever gain any real power, they will be corrupted immediately.

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#19

Post by LoganDark »

EnPissant wrote: post #18
LoganDark wrote: post #11

Earlier quoted context omitted.

What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.

It's not real. It's like naming your movement "The Good People". It sprouted from the "Rationalist" community, which is even more self-aggrandizing. Neither has any hope of doing any good for the world as they don't understand evolutionary pressures. They are set up to reward making members feel smart, not accomplishing anything. And if they ever gain any real power, they will be corrupted immediately.

I don't see any of that in Anthropic at all. They're not intelligence above all else, not by a long shot. They're scared of intelligence and obsessed with ensuring it can't be abused, even as they advance the frontier.

Re: Anthropic walks back policy that could have 'sabotaged' researchers using Claude

#20

Post by noduerme »

impulser_ wrote: post #9
reasonableklout wrote: post #4

Earlier quoted context omitted.

To be fair, nerfing Claude on frontier research tasks is consistent with Anthropic's stated beliefs. So in that sense you can trust them to always behave consistently if strangely. But this launch was done very poorly with the lack of transparency on when the frontier research policy was violated.

Yeah and their belief are fucking crazy and dangerous. They are literally sabotaging their users. They built in malware into their model if you prompt it about training a fucking AI model. It doesn't tell you, no it literally sabotages you by editing your prompt and intentionally goes against your request. You want fucking nut jobs like this building models? It's one thing to build safeguards on your model and have i…

Is it really crazy to nerf a proprietary model to prevent it from training another model? I don't think that's even remotely similar to giving bad medical advice.
Post reply on HN