Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

221–230 of 935 posts

Re: We must pace the frontier

#221

Post by shepherdjerred »

youoy wrote: post #134
pr337h4m wrote: post #16

> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. This is the only concrete prediction in th…

I like to replace thes AI text with "virus manipulation" "Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems." If a CEO of a health company was saying this, the reactions would not be that chill. The worst failure of our society is to call this technology AI, intead of something line "artificial general auto…

A rather unfair comparison.

The whole calculus here is that others are also developing these systems which has led to a race.

A much better comparison to the situation is the nuclear weapons arms race.

Re: We must pace the frontier

#222

Post by InsanityCheck »

anon84873628 wrote: post #43
pr337h4m wrote: post #16

> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. This is the only concrete prediction in th…

Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together? For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.

Currently Qwen3.8 27B is roughly on Opus 4.6 level. In at most a year given the current pace, you could probably run such hacking bot nets out of a reasonably small local server, bootstrapping by hacking or acquiring login credentials for more compute.

Re: We must pace the frontier

#223

Post by Jordan-117 »

Jcampuzano2 wrote: post #63
TheSisb2 wrote: post #38

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction.

We built the paperclip maximizer, and it is capitalism.

Re: We must pace the frontier

#224

Post by shepherdjerred »

anon84873628 wrote: post #43
pr337h4m wrote: post #16

> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. This is the only concrete prediction in th…

Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together? For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.

it’s two-fold. Either malicious actors or the AI systems themselves.

Hugging Face showed that AI can do serious hacking without really being told to. If a model had its own motivations there could be real damage.

Re: We must pace the frontier

#225

Post by baq »

ks2048 wrote: post #209
heaney-555 wrote: post #11

None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.

Nothing says "Let's make a deal" like constantly insisting we are Good and they are Evil.

In politics every single person playing the game is acutely aware of the rules. This is why talks are held behind closed doors so the rules can be suspended for a while.

Re: We must pace the frontier

#226

Post by davemp »

Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.

In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.

At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:

- Internet connected infrastructure being crippled

- Creation of WMDs

- Economic collapse (precipitous devaluation of knowledge work and IP).

I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.

Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).

The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.

We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.

Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.

I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.

We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.

Re: We must pace the frontier

#227

Post by cja »

Can the frontier be paced partly by holding humans responsible for the actions of their software?

My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.

I'm not a lawyer but surely the bad actions of AI are covered by existing law.

I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.

P.S. Strictly, development wouldn't decelerate, but be focused more on safety.

P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.

P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?

Re: We must pace the frontier

#228

Post by baq »

sailfast wrote: post #212
youoy wrote: post #134

Earlier quoted context omitted.

I like to replace thes AI text with "virus manipulation" "Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems." If a CEO of a health company was saying this, the reactions would not be that chill. The worst failure of our society is to call this technology AI, intead of something line "artificial general auto…

Can we build level IV AI containment labs?

No but we can watch AI hack into a BSL4, once

Re: We must pace the frontier

#229

Post by InsanityCheck »

Ydarbleoj wrote: post #199
maxutility wrote: post #179

I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other in…

He talks too much and as a result it's looked like pre-IPO hype and disingenuous because if he believed what he says then he'd stop. Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.

??? I bet investors are extremely happy that anthropic pledged to give an outside company full access to their IP, in order to slow down their own iteration speed. Investors looooove oversight for a company they invest in. I wonder why no other company does stuff like this

Re: We must pace the frontier

#230

Post by stratos123 »

timmg wrote: post #18

I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work. First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world? Maybe this is a ploy for "regulatory capture". And that might help them in the s…

> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them?

The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.

Post reply on HN