Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our position on open-weights models

anthropic.com

Post reply on HN
Print view View original

641–650 of 1001 posts

Re: Our position on open-weights models

#641

Post by 0xDEAFBEAD »

tyre wrote: post #374
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

What are you suggesting as an alternative? Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith. What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.

What if it was an international organization with stakeholders from various national governments and international bodies?

Re: Our position on open-weights models

#642

Post by 0xDEAFBEAD »

kfse wrote: post #638

With respect to whomever this dude thinks he is, it's egregiously "main character" to frame a policy recommendation for the US in terms of what's important to his personal beliefs.

That's just... how political discourse works. You form beliefs about the world, and recommend policies on the basis of your beliefs. That's what everyone is doing when they discuss policy.

Re: Our position on open-weights models

#643

Post by abotsis »

China releasing open weight models, powerful or not, does nothing to prevent their development of models they’ll use for evil. Nor does it stop someone from abliterating a non-Chinese model and using it for evil.

Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.

Re: Our position on open-weights models

#644

Post by sciencejerk »

matheusmoreira wrote: post #468
vhantz wrote: post #196

Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!) The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be…

> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag. Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets disti…

Won't regulators just lock down all hardware? All bootloaders and OS are signed?

Re: Our position on open-weights models

#645

Post by 0xDEAFBEAD »

InkCanon wrote: post #625
rramach wrote: post #358

Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator. If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk. One may or may not agree with item 1 but item 2 above should have broad support given the unkno…

This argument is old and goes back to cyber security. Things like government backdoors are the same. It's true centralizing power in the hands of a "good" would theoretically be good. But the key part is the "good". I think after repeated experiences with governments and corporations, no rational person would ever assume them to be good. There are simply too many incentives to be bad.

"We can't trust the government to control nukes! Every citizen should be able to build a nuclear weapon in their backyard with zero regulatory authorization!"

Re: Our position on open-weights models

#646

Post by Stitch4223 »

$regulatory_capture

https://en.wikipedia.org/wiki/Regulatory_capture

On processing power, copyright, and capability.

I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.

Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).

Edit: spelling

Re: Our position on open-weights models

#647

Post by alevskaya »

ajyoon wrote: post #172

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be a…

I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...

The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.

Re: Our position on open-weights models

#648

Post by CMay »

vhantz wrote: post #196

Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!) The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be…

> The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.

If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.

> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips

Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.

Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.

In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.

Re: Our position on open-weights models

#649

Post by tesnorindian »

AI should not be centralized and should be distributed. There can be no 1 provider to serve all of our needs. We would require both Open weights and Closed weights models for a lot of our use cases.

Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.

More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.

Re: Our position on open-weights models

#650

Post by stratos123 »

elliotec wrote: post #391
rramach wrote: post #358

Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator. If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk. One may or may not agree with item 1 but item 2 above should have broad support given the unkno…

Closed-weight models can also pose biological risk, arguably more so than open-weight models, given the fact that they're being used by state actors (the United States military) to kill people right now. Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and wa…

> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?

I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.

> And the rest of us have no way to stop it or defend.

If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.

Post reply on HN