Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our position on open-weights models

anthropic.com

Post reply on HN
Print view View original

461–470 of 1001 posts

Re: Our position on open-weights models

#461

Post by orbital-decay »

mjorgers wrote: post #16

So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology. Quis custodiet ipsos custodes?

There's more to it than just safety: in Dario's psychopathic mind, obtaining advanced AI will lead to massive military advantage, hence "we" must do it first to prevent "them" attacking "us". See where it's going? Suppose it happened, then what? By Dario's logic there are two possible continuations: 1. start a war by attacking first and 2. do nothing and waste the entire effort because "they" will obtain the same capability a bit later.

Re: Our position on open-weights models

#462

Post by monk_grilla »

abacadaba wrote: post #428
soundworlds wrote: post #257

What Dario misses time and time again, is that people don't trust the US to create aligned AI anymore. His entire strategy rests on the assumption that the US (and their government) are exceptional. This is clearly false to the rest of the world.

then we will make them trust us

The most American thing I've ever heard lmao

Re: Our position on open-weights models

#463

Post by niklasrde »

usef- wrote: post #453
GodelNumbering wrote: post #155

In the first paragraph, > Anyone who has read my past writing should know that I don’t regard such bans as a useful measure, Later (on banning chip sales to china) > we should crack down on the rampant smuggling and workarounds used to obtain access to such chips. If you truly believe that bans don't work, the same applies to hardware too. Furthermore, Dario says later "To address these concerns, I do support the fol…

Banning software/data from being put on the internet vs banning GPU hardware is fundamentally different though, isn't it?

How so?

Re: Our position on open-weights models

#464

Post by jackdeansmith »

bigyabai wrote: post #387
credit_guy wrote: post #370

I know it's unpopular, or unfashionable, but I agree with this letter. LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle. It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. M…

> We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle. If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.

>If the biggest danger of LLMs is that they can hack traditional systems

This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.

Re: Our position on open-weights models

#465

Post by marcus_holmes »

dustin_vk wrote: post #375
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

Yeah this is bad. I'm cancelling my Claude subscription and I'd encourage everyone else to do so too.

I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money.

It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.

Re: Our position on open-weights models

#466

Post by jjfoooo4 »

bryan0 wrote: post #362
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

> Yeah, this is anthropic advocating for a ban on open weight models. This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting…

> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution

How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.

Would this restrict downloads of an open model, or publishing?

Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?

Re: Our position on open-weights models

#467

Post by usef- »

kelnos wrote: post #321
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

Or the important question: what happens if the model fails this test? Presumably then it gets banned; otherwise what's the point of the test if no action is taken if it fails? More self-serving trash from the US AI companies, disguised as "being reasonable".

It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.

Re: Our position on open-weights models

#468

Post by matheusmoreira »

vhantz wrote: post #196

Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!) The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be…

> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.

Re: Our position on open-weights models

#469

Post by nmfisher »

monk_grilla wrote: post #417
m3h wrote: post #207

Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country. No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".

> Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country... I might have missed something but wasn't the big story that Dario refused the Department of War's demand to use Anthropic's models for such purposes?

He was directly asked in a Bloomberg interview whether Claude was used in the bombing of the girl’s school in Iran, and his answer was “We don’t know”.

His redline was autonomous weapons, not the death of 100 innocent girls.

Re: Our position on open-weights models

#470

Post by p1necone »

coffeemug wrote: post #176
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

A government agency tests all medications, why not models?

Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.

If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.

Post reply on HN