We must pace the frontier

> We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top

> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time

And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.

I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?

Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?

> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.

This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.

Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.

> A race to the bottom, spurred by commercial incentives, can make these risks more acute

I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.

If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.

Well a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.

Do you realize how big "the entire internet" is?

> How many dollars of compute do you believe were available to the swarm(s)

At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)