Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

561–570 of 934 posts

Re: We must pace the frontier

#561

Post by nedruod »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.

You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.

The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.

We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.

When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.

Re: We must pace the frontier

#562

Post by Applejinx »

vanviegen wrote: post #512
mullingitover wrote: post #385

Earlier quoted context omitted.

> The best models are owned by a few companies Whenever I see 'best model,' 'most advanced model,' etc, it just reads to me as 'roundest ball'. The new model is the most precisely round ball ever, models next year are going to be even rounder , etc. The difference in utility between the latest, most round ball and last year's frontier balls (which are now freely available to the public) is pretty debatable, imo.

Making balls ever rounder clearly has diminishing returns, as there's a limit to roundness. Is there a similar limit to intelligence though?

Yes, absolutely. Even human intelligence stops being useful at a certain point and becomes 'enough, provided the human is motivated and tenacious'. Intention is a better metric, but it's a serious AI weak point if not actively an achilles' heel. 'Warios' come to mind.

Re: We must pace the frontier

#563

Post by bennydog224 »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

I agree it’s not all altrusim. It’s a little less clear what you mean at the end though.

For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?

Re: We must pace the frontier

#564

Post by dismalaf »

baobabKoodaa wrote: post #502
dismalaf wrote: post #409

This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent). If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If…

You people just never stop, do you? At some point you're gonna have to look back at all the times you said "LLMs have hit a wall" and look at what the progress was since the last time you said that. Please do go ahead and revisit your comment here 1 year later. Embarrassing.

Why don't you actually read the words and apply a little critical thinking?

How often do you hack on actual LLMs? Or do you just use the chatbot or API for your agents? An LLM without internet access or tools is just as useless as a year ago.

Re: We must pace the frontier

#565

Post by estearum »

fwn wrote: post #509
twoodfin wrote: post #483

Earlier quoted context omitted.

You believe essential infrastructure is air-gapped from the public internet today?

No. And no alignment, guardrails or slowdowns will ever secure our insecure infrastructure. Bad infrastructure decisions are a risk independent from AI. The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status. Cybersecurity is such an important topic for a country, dreaming about global alignment ju…

> Bad infrastructure decisions are a risk independent from AI.

All decisions are made relative to their tradeoffs including costs and risks.

"Water levels rising 100x beyond historical levels are not the problem. The dam's height is the problem!"

All systems should be made infinitely secure and all dams should be made infinitely tall.

Re: We must pace the frontier

#566

Post by tfehring »

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.

Re: We must pace the frontier

#568

Post by neomantra »

baobabKoodaa wrote: post #458
neomantra wrote: post #375

[In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons: * multiple levels of inappropriate controls and unintended consequences in several complex systems * the inability, both politically and technically, to turn it off [1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf EDIT: Comments in…

Your link is from 2013. You're intentionally using the term "Agent" to muddy the waters. Don't do that.

Edited for clarity, thank you. I wasn't trying to muddy the waters (doesn't mean I didn't), it's a term I've used for a long time. I did mistakenly think the quotes would help disambiguate. I don't know what Knight called them; the first S in "SMARS" is Smart.

Re: We must pace the frontier

#569

Post by pizza234 »

hgoel wrote: post #554
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources? The companies doing these things without following common sense security measures are the felony generators.

> the only incidences of LLM generated felonies involved misconfigured sandboxes

This is false; see the analyses of the latest incidents.

Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.

And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.

Re: We must pace the frontier

#570

Post by pizzly »

I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you.

I don't think I would like this future.

Post reply on HN