Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our position on open-weights models

anthropic.com

Post reply on HN
Print view View original

491–500 of 1001 posts

Re: Our position on open-weights models

#491

Post by paxys »

rubslopes wrote: post #367
ajyoon wrote: post #172

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be a…

If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.

Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.

Re: Our position on open-weights models

#492

Post by [deleted] »

artrockalter wrote: post #187
ajyoon wrote: post #172

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be a…

The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist…

[deleted]

Re: Our position on open-weights models

#493

Post by cogman10 »

mrcwinn wrote: post #411
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

So, if an open weights model was found to be very dangerous, what - just too bad? One could, of course, design an open safety protocol, written and performed by people in the executive branch, accountable to an elected official. I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is…

> So, if an open weights model was found to be very dangerous, what - just too bad?

Yeah, it's too bad.

I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.

But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.

And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.

Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.

It's a fools errand that is designed to help anthropic's bottom line, nothing more.

Re: Our position on open-weights models

#495

Post by urams »

adastra22 wrote: post #420
ajyoon wrote: post #172

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be a…

The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight. Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.

Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.

What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.

Re: Our position on open-weights models

#496

Post by doom2 »

erwald wrote: post #315
GodelNumbering wrote: post #155

In the first paragraph, > Anyone who has read my past writing should know that I don’t regard such bans as a useful measure, Later (on banning chip sales to china) > we should crack down on the rampant smuggling and workarounds used to obtain access to such chips. If you truly believe that bans don't work, the same applies to hardware too. Furthermore, Dario says later "To address these concerns, I do support the fol…

Bans on Chinese open weight models being used in the US and bans on AI chips and semiconductor manufacturing equipment being exported to China are two extremely different things, and it's not inconsistent in any way to oppose one and endorse another.

> bans on AI chips and semiconductor manufacturing equipment being exported

In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.

Re: Our position on open-weights models

#497

Post by pibaker »

timpera wrote: post #143
badatnames wrote: post #124

I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due…

Anthropic has been surprisingly bad at comms for the last few months, which I find crazy in such a competitive market.

I feel everyone running these companies has stopped caring what the public think. It's all about selling their vision to politicians now. Which is also why they try to tie everything with "national security" — the average joe hears it and thinks about forever wars in the Middle East, the average congressman hears it and approves whatever solution is suggested.

Re: Our position on open-weights models

#500

Post by cloverich »

claaams wrote: post #444
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

Won't this just incentivize companies to move operations outside of the US where these models aren't regulated?

In the near term thats something that can be controlled. If you buy Anthropics take, then even buying time would be a win.
Post reply on HN