We must pace the frontier

This is exactly my view and it's frustrating that a sizeable group of people refuse to accept there's any risk with AI.

We're literally trying to build something that's smarter than all humans and with the capacity to do extraordinary work while most people think there's no risk in open sourcing this capability with no guardrails (model ablation means no guardrails and as far as I'm aware can't be stopped).

Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.

That claim looks to be coming from the opposite of Occam's Razor. Them doing typical SV dirty plays is a lot more easier explanation than the straws you are grasping at.

Buckle up.

I’ve spent a lot of time studying opinions coming out of the frontier labs, and talking directly to employees. I think we’re cooked for the following reasons

- Everything connected to the Internet will eventually be hacked. For decades, we’ve slowly migrated all of our critical systems to devices connected to one giant network. Misaligned agents have mounted increasingly sophisticated explorations/attacks by infiltrating systems that are completely unprepared to deal with this kind of threat.

- We have a poor understanding of the impact of LLMs, which is inhibiting our ability to protect ourselves, create the right guardrails, and stand up policies quickly enough. We’re still wasting time debating. We’re moving quickly toward a punctuated equilibrium moment and won’t have time to waste

- We’re at the beginning of a runaway evolutionary process. At some point soon, someone is going to invent recursive self improvement. Any policy will at best serve as a mild disincentive. It’s happening.

- In the very best case, someone will exploit the confusion and chaos of this situation for their own gain. There are many examples littered throughout history, including the last several years of course. The utopian vision is not happening. Like all other resources, we’ll compete. We haven’t devised a good approach to orderly distribution of resources for large groups of people.

- No one has demonstrated that they understand how to build AI safely. Not a single entity. These companies are not even being scaled in a way that’s conducive to building AI safety. Hyperscale orgs struggle with scaling basic consumer apps. How on earth are they going to scale AI? Based on my firsthand observations of their internal workings, I have zero confidence they can.

This won’t necessarily