Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our position on open-weights models

anthropic.com

Post reply on HN
Print view View original

981–990 of 1001 posts

Re: Our position on open-weights models

#981

Post by throw1234567891 »

rramach wrote: post #358

Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator. If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk. One may or may not agree with item 1 but item 2 above should have broad support given the unkno…

> but item 2 above should have broad support given the unknown unknowns in play

We have a term for that: FUD.

Re: Our position on open-weights models

#982

Post by thaway7388 »

0xDEAFBEAD wrote: post #663
andix wrote: post #302

China doesn't seem to think that powerful open weight models are a serious threat to them. Otherwise they wouldn't release them. Those models could also be used by their enemies against China. I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.

China is worried about AI. In fact, Chinese leadership has expressed more worry than US leadership. "Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and e…

> "ensure that AI is always under human control"

He doesn't say "our own control". AI security is a very serious matter. The way it's being discussed right now gives more harm than benefit to understand the problem.

This is because it's discussed in Cold War/WW3 context. This is exactly Dario's mentality in this announcement. And it's understandable since USA is a country actively fighting in constant wars. They even have a Department of War.

This is unlike the rest of the world which have more like a Peace-time mentality. In peace times, trade, collaboration and good relationships are more important than competition or arms race. China seems like playing this game better than US at the moment.

Dario also speculates to take the potential benefits or dangers of AI to the extremes. This is also not very healthy thinking and can be dangerous when combined with war mentality.

Re: Our position on open-weights models

#985

Post by FinchNova12 »

jackcviers3 wrote: post #930
tcldr wrote: post #919

Ban distillation of our outputs, but our distillation of the sum-total of civilisation's intellectual output – proprietary or otherwise – is fair use? Either everyone licenses, or nobody does. And if you can't enforce licensing bans for everyone, the de-facto loser is those who you'd probably want to support the most, start-ups and universities, while your adversaries gain the upper hand. The strongest argument for r…

On the other hand, the open weights models could crawl and annotate and rl the training data that Anthropic and OAI did in exactly the same way, and take the exact same legal hits. They use distillation because it's cheaper not to do so.

Hmm, why do you think this is true? One reason I'm skeptical of this is because RL envs are often purchased (and are not publicly available), and this might be a sizable component of why models are getting better.

Re: Our position on open-weights models

#986

Post by alok-g »

lenerdenator wrote: post #473
vhantz wrote: post #196

Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!) The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be…

Honest question: If your business model both produces the SOTA for something and isn't profitable, is the price too high, though? While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them. Really, thi…

>> If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?

The market dynamics would find good balance automatically Meanwhile, good market practices and fair competition should be continued.

Re: Our position on open-weights models

#987

Post by FloorEgg »

I struggle to understand how they can frame for the greater good to be anti-distillation and pro-open-weight-models.

I can understand how they would be against competitors distilling their models and for having access to competitor open weight models.

At first glance it seems like a pragmatic answer: "banning open weight model use by US companies doesn't help", but to what question? What if those open weight models are poisoned and primed to create backdoors in US companies? If the government has intelligence this is a credible risk, what should they do about it?

Overall it feels like the letter is conflating a bunch of motives, some of which may be opaque, and at face value doesn't seem logically consistent.

Re: Our position on open-weights models

#989

Post by pejrich »

areoform wrote: post #186
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

When Fable was yanked, it was said to be (in part) due to the "jailbreak" of instructing Fable to "fix this code" — https://news.ycombinator.com/item?id=48552687 Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code? There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth. Why not just give…

It's not that simple. The odds are always stacked in favor of the hacker. It's like saying, "why not just make a prison that's impossible to escape from". You can make a prison very, very hard to escape from, but you have to shut off every possible way someone could try to escape, whereas someone trying to escape only has to find one vulnerability, once. It's much harder to plug every possible hole in a complex system than it is to find one point of weakness.

Re: Our position on open-weights models

#990

Post by pejrich »

tyre wrote: post #374
cogman10 wrote: post #28

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…

What are you suggesting as an alternative? Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith. What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.

Any model that is going to be meaningfully useful is going to have the potential for harm too. It's not possible to know enough about chemistry to be useful to a chemist, without also being able to figure out how to synthesize drugs. It's not possible to have enough knowledge to be useful to any/all programmers, without being able to figure out how to hack a system, or create a virus. That's just kind of life though.
Post reply on HN