Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

251–260 of 942 posts

Re: We must pace the frontier

#251

Post by timmg »

stratos123 wrote: post #230
timmg wrote: post #18

I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work. First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world? Maybe this is a ploy for "regulatory capture". And that might help them in the s…

> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.

Well: do they listen to the evaluators?

I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.

The question then is: do we stop when the safety people say to (they will) or not?

Re: We must pace the frontier

#252

Post by stratos123 »

Ydarbleoj wrote: post #199
maxutility wrote: post #179

I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other in…

He talks too much and as a result it's looked like pre-IPO hype and disingenuous because if he believed what he says then he'd stop. Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.

> disingenuous because if he believed what he says then he'd stop.

Why do you think so?

Re: We must pace the frontier

#253

Post by adrianN »

spidersouris wrote: post #52
TheSisb2 wrote: post #38

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…

The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?

We‘d have to occasionally bomb all computing infrastructure in other countries to prevent them from training.

Re: We must pace the frontier

#254

Post by kalkin »

nullbio wrote: post #241
cja wrote: post #227

Can the frontier be paced partly by holding humans responsible for the actions of their software? My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused. I'm not a lawyer but surely the bad actions of AI are covered by existing law. I suspect that the development of AI would dece…

That's the -only- way it should be paced. But then regulatory capture wouldn't happen, so of course they won't suggest this one.

I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't.

- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.

- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.

Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...

To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.

Re: We must pace the frontier

#255

Post by pastel8739 »

Razengan wrote: post #148
TheSisb2 wrote: post #38

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…

> genuinely afraid of the inevitability of AI turning into I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence? "durrr it's just matrix multiplications" mfer so is your brain. What humans should be doing is overhauling archaic social institutions to keep u…

Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.

Re: We must pace the frontier

#256

Post by nullbio »

0xDEAFBEAD wrote: post #250
nullbio wrote: post #239

Earlier quoted context omitted.

And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign. It's all so obvious.

Who would you pick for "embedded evaluator"? "deep ties to everyone in the doomer media campaign" Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate. https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:... At a certain point, Occam's Razor says: These engineers are legitimately worried. They…

Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs.

Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.

Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.

Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.

Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.

They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.

Re: We must pace the frontier

#257

Post by stratos123 »

swingboy wrote: post #130

There’s a difference between a model recursively improving “itself” and improving itself via online learning, right? The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session). The latter seems far…

The latter has some extra failure modes but I don't think these two kinds of RSI are that different. Either way you can get exponential growth in capabilities, and either way a not-entirely-aligned model can train a more capable and more misaligned successor.

Re: We must pace the frontier

#258

Post by pastel8739 »

iloveoof wrote: post #75

Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.

He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine

Re: We must pace the frontier

#259

Post by zer00eyz »

p1esk wrote: post #127
zer00eyz wrote: post #55

Earlier quoted context omitted.

> Hypereautomation will kill jobs and destroy the economy. What do you think that were going to hyper automate? Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work? All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script,…

Meanwhile in the real world hundreds of billions are being invested into humanoid robot development. In 2-3 years my Optimus 4 will do all the gardening and plumbing I need.

Fusion Reactors, Flying Cars, Self Driving Cars...

We're really bad about predicting the future.

Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.

Post reply on HN