Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

681–690 of 950 posts

Re: We must pace the frontier

#681

Post by CamperBob2 »

aswegs8 wrote: post #601
hgoel wrote: post #586

Earlier quoted context omitted.

The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the inc…

There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.

Letting bad actors dictate the pace of technological development is certainly one option, but not a good one.

Re: We must pace the frontier

#684

Post by bobogei81123 »

I don’t claim this is their entire motiv, and I’m sure plenty of people internally are genuinely concerned about AI risks and the disruptions caused by moving too fast. But I can’t help but feel that part of the push to "pace the frontier" is an attempt to form a "Phoebus Cartel" for AI labs.

Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.

We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.

Re: We must pace the frontier

#685

Post by [deleted] »

globnomulous wrote: post #665
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

> RSI For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."

[deleted]

Re: We must pace the frontier

#687

Post by gverrilla »

ethagnawl wrote: post #624

> I believe that AI could cure most major diseases in the next 5–10 years This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)

Why would you expect anything different from the CEO of a company that's going to IPO?

Re: We must pace the frontier

#689

Post by clivefx »

hgoel wrote: post #586
pizza234 wrote: post #569

Earlier quoted context omitted.

> the only incidences of LLM generated felonies involved misconfigured sandboxes This is false; see the analyses of the latest incidents. Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given. And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even gett…

The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the inc…

What company, product, or period of industrial history do you think met your standard of prudence?

Re: We must pace the frontier

#690

Post by greatgib »

pizza234 wrote: post #569
hgoel wrote: post #554

Earlier quoted context omitted.

Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources? The companies doing these things without following common sense security measures are the felony generators.

> the only incidences of LLM generated felonies involved misconfigured sandboxes This is false; see the analyses of the latest incidents. Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given. And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even gett…

"the rules they had been given".

Remember, they are just algorithms. You pull the plug and there is no light anymore

It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.

And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.

Post reply on HN