Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

431–440 of 943 posts

Re: We must pace the frontier

#431

Post by bennydog224 »

Disclaimer: I’m in security engineering and thus I’m primed to being skeptical. I am in full agreement that security assessments and regulation certainly needs to be stricter in foundation labs. However, I’m extremely skeptical when Amodei, Altman, and Musk all agree on something at the same time as it’s becoming viral and hits mainstream media.

Red flags:

- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.

- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.

- These labs would all benefit from nationalization if their funding model fails (an easy bailout).

- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.

- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.

- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).

Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.

Re: We must pace the frontier

#433

Post by stratos123 »

youoy wrote: post #341
stratos123 wrote: post #276

Earlier quoted context omitted.

Okay, but suppose that Hypothetical Opensource Anthropic trains a model that turns out to be very dangerous, and releases it. Suppose that the public investigates and, not being limited by an information bubble, correctly notices that it's very dangerous. What then? The model's already released, there's effectively no way to prevent it from being used. Whatever the risks of its release were, they will now materialize…

By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other…

> By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic.

It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.

However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.

Re: We must pace the frontier

#434

Post by Loquebantur »

Aurornis wrote: post #397
Chance-Device wrote: post #306

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. Regulate what, exactly? Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules. There’s a theoretical version where all the countries in the world join hands and agree not to compete…

You start from incorrect premises by presupposing agreed-upon rules would never work anyway.

"Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to.

Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.

Re: We must pace the frontier

#435

Post by PantaloonFlames »

gewa wrote: post #91
Jcampuzano2 wrote: post #63

Earlier quoted context omitted.

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

We are still living in a capitalistic society. We have to find a solution which is responsible, safe and returns on the investment. The commercial aspect can be true at the same time.

The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.

Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?

Re: We must pace the frontier

#436

Post by MachineMan »

Its not the model, it is the compute that is the bottleneck. Dario says one thing but does the other as he scales energy and compute. It is and has been known that it will be ai that will be blamed for a wipe, that will be coordinated by human actors. If he really wants to not hurt people, he would shut it down. This blog post is therefore a classic case of covering his behind.

Re: We must pace the frontier

#437

Post by MaKey »

maxutility wrote: post #179

I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other in…

> I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing.

Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?

Re: We must pace the frontier

#438

Post by p1esk »

zer00eyz wrote: post #259
p1esk wrote: post #127

Earlier quoted context omitted.

Meanwhile in the real world hundreds of billions are being invested into humanoid robot development. In 2-3 years my Optimus 4 will do all the gardening and plumbing I need.

Fusion Reactors, Flying Cars, Self Driving Cars... We're really bad about predicting the future. Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.

We have nuclear reactors, helicopters, and self driving cars already. And we finally have powerful AI that we could only dream about just 5 years ago.

We now have everything we need to scale up humanoid robot training: algorithms, hardware, and money to buy a lot of training data/compute. Fierce competition and strong economic motivation will force rapid progress in this field.

Re: We must pace the frontier

#439

Post by xg15 »

jfreds wrote: post #400
rickydroll wrote: post #372

I also like the idea of pacing the frontier and slowing down development so we all have a chance to stop and catch our breath. However, I don't think regulating AI is the way to do it. I would tackle the problem by constraining the resources used to run AI models; the "simplest" way to do that is increasing the costs of running a data center. The two easiest places to raise data center costs are electricity and water…

I think taxes like these actually worsen the incentive structure. If it’s effectively more expensive to do training and inference, labs will be incentivized even more to improve efficiency. Opaque recurrence seems like a great way to improve efficiency, and it comes at the cost of safety

Even more opaque than the models already are? I take that risk.

I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...

Re: We must pace the frontier

#440

Post by PantaloonFlames »

credit_guy wrote: post #217
Jcampuzano2 wrote: post #63

Earlier quoted context omitted.

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.

That level of revenue is apparently not enough for them to feel confident in the stability of their competitive position.
Post reply on HN