Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

991–1000 of 1001 posts

Re: We must pace the frontier

#991

Post by kvuj »

throw10920 wrote: post #989
cuuupid wrote: post #435

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

> - no open weights That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free. > - can’t use claude to research AI No problems there either. > - train on everyone else’s IP and sell it back to them Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal. > - 8 regulatory capture attempts and coun…

> That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.

But the Chinese are doing just that?

> Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.

Legal != moral

Re: We must pace the frontier

#992

Post by kimseungyong »

The problem is that, as seen in the case of the Houthi rebels developing missile software using Claude Code, AI has lowered the very threshold at which human cruelty or selfishness can be put into practice. Anthropic’s safeguards were applied on a per-request basis, but splitting the work into multiple sessions prevented me from seeing the bigger picture. After all, human instinct does not transform into technology, so the only remaining way is to pile up the friction that takes that instinct to turn into action all over again.

However, the international agreement proposed by Dario Amodei is ultimately nothing more than a promise made by people. There is also a precedent in which the Nuclear Non-Proliferation Treaty could have simply withdrawn if it had been cornered like North Korea. Isn’t the AI speed-control agreement also vulnerable to being broken in the same way if one side finds itself politically cornered. In particular, unlike nuclear weapons, AI has the characteristic of recursively improving itself, allowing its capabilities to accelerate itself, so the moment one side breaks away, the gap can widen in an instant, making the incentive for deviation even greater.

Re: We must pace the frontier

#993

Post by boshalfoshal »

eknkc wrote: post #870
boshalfoshal wrote: post #859

Earlier quoted context omitted.

I think you fundamentally misunderstand what kind of company Anthropic is. I'm simply taking the path of least resistance when understanding what people at Anthropic have to say, and basically every public source points to Anthropic talking the talk as well as walking the walk with regards to AI safety and their ideas on AI governance. Re the point on returning shareholder value, Anthropic is a PBC: > Public Benefit…

Oh I am terribly sorry, did not know Anthropic was formed as a a “Good Company”. Which changes everything ever. Snark aside, maybe I’m too pessimistic and you are correct on this. But.. doesn’t that mean we are completely fucked? If they don’t have any ulterior motive and they truly are panicking.. We should start packing.

yes that is literally why they are sounding the alarm - because people think AI is a party trick and they aren't taking it seriously enough.

Re: We must pace the frontier

#994

Post by boshalfoshal »

qlte wrote: post #982
boshalfoshal wrote: post #859

Earlier quoted context omitted.

I think you fundamentally misunderstand what kind of company Anthropic is. I'm simply taking the path of least resistance when understanding what people at Anthropic have to say, and basically every public source points to Anthropic talking the talk as well as walking the walk with regards to AI safety and their ideas on AI governance. Re the point on returning shareholder value, Anthropic is a PBC: > Public Benefit…

The decision to found Anthropic to compete with OpenAI's then concentration of SOTA LLM AI researchers, and then achieving/surpassing parity with OpenAI is pretty much directly responsible for creating the current capabilities arms race. Google has shown they are not motivated at all to push the bigger and bigger SOTA frontier with Mythos-type super models. Even OpenAI isn't really even trying to surpass Anthropic at…

If you've read enough lesswrong, this won't be new to you. The strategy always was "become the most capable model lab so that you have leverage to control AI development."

Re: We must pace the frontier

#995

Post by throw10920 »

kvuj wrote: post #991
throw10920 wrote: post #989

Earlier quoted context omitted.

> - no open weights That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free. > - can’t use claude to research AI No problems there either. > - train on everyone else’s IP and sell it back to them Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal. > - 8 regulatory capture attempts and coun…

> That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free. But the Chinese are doing just that? > Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal. Legal != moral

> But the Chinese are doing just that?

Completely and totally irrelevant. Has nothing to do with the fact that you are not obligated to others' work.

> Legal != moral

Sure, you're right.

I guess the bigger issue is that the GP's post calls out Anthropic for following along in legal (but what we both believe to be immoral) mass copyright infringement that OpenAI started, while absolving them of the exact same thing. It's a shill account.

Re: We must pace the frontier

#996

Post by emodendroket »

RGS1811 wrote: post #495

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

You don't see this as an attempt to create a new regulatory moat then?

Re: We must pace the frontier

#997

Post by rphv »

glub wrote: post #289

Dario, Sam, and Elon are all on the same page on this. So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know? OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models…

> So it's either they truly think AI is going to kill us all

Yes. Basically everyone here is dismissing this possibility. We shouldn’t.

Re: We must pace the frontier

#998

Post by ta20240528 »

glub wrote: post #289

Dario, Sam, and Elon are all on the same page on this. So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know? OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models…

"So it's either they truly think AI is going to kill us all"

Exactly how does a CPU kill 8 odd billion people? Even if they got hold of the weapons to wipe out a few billion, surely the remain billions would just turn off the data centres?

Re: We must pace the frontier

#999

Post by maxnevermind »

Hmm, what a coincidence both OpenAI and Anthropic suggest to slow down. I think the signal it sends in plain English sounds something like: We burn too much money atm and shit is about to hit the fan with all that compute capacity starting to rapidly come online 2027, we are not seeing enough demand to justify all of those data centers we already agreed to pay for, let's do a damage control and lower expectation of the plebs.

Re: We must pace the frontier

#1000

Post by aryehof »

RGS1811 wrote: post #495

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

> Without alignment, further improvements in capability turn LLMs into wanton felony generators

Isn’t “full” alignment and guardrails a task that can never be generally achieved? One will always discover and realize the need for new guidelines and guardrails? What about vague, incomplete, inconsistent nuances and generalizations - that make this a losing battle?

I suggest ultimate safeguards need to be outside the models?

Post reply on HN