Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our position on open-weights models

anthropic.com

Post reply on HN
Print view View original

971–980 of 1001 posts

Re: Our position on open-weights models

#971

Post by anthonypasq »

chorizo wrote: post #628
GodelNumbering wrote: post #155

In the first paragraph, > Anyone who has read my past writing should know that I don’t regard such bans as a useful measure, Later (on banning chip sales to china) > we should crack down on the rampant smuggling and workarounds used to obtain access to such chips. If you truly believe that bans don't work, the same applies to hardware too. Furthermore, Dario says later "To address these concerns, I do support the fol…

At this point, Anthropic releasing a sonnet 4.x class open weights model would be a positive PR move that doesn’t affect their bottom line.

They will never do this because it will reveal way more about their other model architectures than they would ever be willing to do. I mean really why the hell are we mad at a company for not wanting to open source their IP? do you think Meta is going to release their ad targeting algorithms? Or google is going to release their search rank algorithms. i mean these things are multi trillion dollar pieces of IP.

I get Anthropic is being really sleazy and annoying here, but being mad at them for not releasing an open weight model i think is unreasonable.

Re: Our position on open-weights models

#972

Post by Majromax »

YmiYugy wrote: post #202
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of op…

> Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.

Worse than that: an open-weight but safe model can be 'abliterated' to remove safety refusals using fine-tuning procedures that require a couple of orders of magnitude less compute than the original pretraining.

The 'universal evaluation' criterion then has three outcomes:

* It could become a mandatory, regulatory oversight of _all_ model training capable of hosting frontier-scale models. Since GPUs for LLM training are the same GPUs for other model training, effective mandate would require GPUs be government owned or controlled as if they were weapons of mass destruction.

* It could impose limits on release of capable open-weight models, requiring Kimi et al to prove that they cannot be made capable of abusive behaviours.

* It could be security theatre.

The AI-as-existential-risk argument points towards the first, the competition-protection argument points towards the second, and least-effort implementation would be the last.

Re: Our position on open-weights models

#974

Post by jmull »

I'm a lot more worried about the US government than China's -- it has a lot more direct impact on my life and is largely controlled by billionaires who do not have good intentions toward the rest of us.

And, to me, this letter comes off as quite insincere. Stopping distillation can only be explained as an anti-competitive measure. Their own explanation is nonsensical -- they say is needs to be stopped to help prevent authoritarian governments from overtaking the US at the frontier of AI. But by its nature distillation lags behind the frontier. Not to mention the US is one of the authoritarian governments we need to be concerned with, and the next thing they advocate for is full, worldwide regulatory control of AI, which is rather heavily authoritarian.

These guys are making a $T gamble and need to screw over a lot of people very badly to make it pay off. You do not want to trust anything they say.

Re: Our position on open-weights models

#976

Post by Stitch4223 »

ianm218 wrote: post #874
Stitch4223 wrote: post #646

$regulatory_capture https://en.wikipedia.org/wiki/Regulatory_capture On processing power, copyright, and capability. I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV. Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not w…

Where does the bio risk he mentions fall in this category? It feels quite plausible in a year or two to have closed loop labs with very modest resources. Doesn’t mean that you have to think only Anthropic should be able to make bioweapons or whatever, but it just feels like this ignores the risk

The bio argument is a call for guardrails, thus regulatory capture.

Using a technology does not make an illegal activity illegal. It already was illegal to begin with.

And there are already guardrails in the form of ethos, law, justice departments and so on. This reality is flagrantly dismissed in their position on open-weights.

Just like cyber crime is just things like extortion etc which are already illegal: following similar reasoning all computer activity should be regulated by their vendors.

Hence the knives factory analogy, which is even more basic to point out this crooked way of reasoning.

Re: Our position on open-weights models

#977

Post by throw1234567891 »

paxys wrote: post #349
badatnames wrote: post #124

I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due…

Anthropic has always been like this. People just responded to the message better when it came from the quirky underdog rather than the trillion dollar behemoth.

Trillion dollar behemoth, heheh.

Re: Our position on open-weights models

#978

Post by hmokiguess »

0xDEAFBEAD wrote: post #654
hmokiguess wrote: post #101

So your concern is safety, and you claim you are the only one that can give us safety but do so by keeping your product closed? Then how about you release the weights? I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.

"You claim you're the only one who can keep us safe from nukes. Where's your nuclear weapon blueprint then, huh??"

Comparing this to a nuclear weapon is funny, but sure, let's go there.

I would like to see you try to build and deploy a nuclear weapon campaign without getting noticed, you would not even be able to source the materials or get very far.

Transparency alone does not do anything, if you rely just on secrecy to protect yourself you are already extremely vulnerable. Deploying a weapon is a totally separate undertaking.

Re: Our position on open-weights models

#979

Post by Majromax »

areoform wrote: post #186
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

When Fable was yanked, it was said to be (in part) due to the "jailbreak" of instructing Fable to "fix this code" — https://news.ycombinator.com/item?id=48552687 Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code? There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth. Why not just give…

> Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?

In the specific case of cybersecurity, this is a reasonable medium-term outcome. IMO, the cybersecurity risk is akin to the spread of a disease among an 'immune-naive' group: we can suddenly deploy much stronger attack-finding tools against large, established codebases created with much weaker security designs. The path from here to there will be rough, but it's still fundamentally easier to write secure code than it is to exploit vulnerabilities. (It's just easier yet to write insecure code, giving our status quo problem.)

For other 'safety' matters, defense isn't so easy because the attack and target are so different. An AI propaganda bot or catfisher 'attacks' slowly-evolving human culture; one that instructs on explosives or bioterrorism directly interacts with an accomplice and not a victim. If you believe that knowledge on how to build a pipe-bomb must be restricted, then giving everyone access to Fable does not mitigate the risk.

The controversial limit of this attitude is recursive self improvement and an AI singularity with potentially destructive results. Proponents of this view think that sufficiently powerful AI is risky in nearly unimaginable ways such that the capability itself is harmful. This is part (but not all) of why Fable (originally?) degraded itself when apparently assisting with AI research.

Post reply on HN