Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

981–990 of 1001 posts

Re: We must pace the frontier

#981

Post by bennydog224 »

RGS1811 wrote: post #957
bennydog224 wrote: post #531

Earlier quoted context omitted.

I agree it’s not all altrusim. It’s a little less clear what you mean at the end though. For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?

No, my argument is that the need to pace the frontier means they cannot safely advance in capability due to liability concerns. The competition is already almost caught up. If OpenAI/Anthropic have hit an upper bound on safe capability improvement, the gap will close all the way, and we will have reached the full commoditization of LLM tokens very soon.

Say this is true (we are near or at the upper bound of safe capabilities and tokens are commodities). If they successful get regulated, all they’ll get is a little extra time. The market will soon realize this - regulated or not - and pull investment.

Seems like an excessive announcement just to get a little extra time.

Re: We must pace the frontier

#982

Post by pickleRick243 »

zahlman wrote: post #971
roncesvalles wrote: post #907

Earlier quoted context omitted.

"Write a blog article about . Keep rewriting it until it passes Pangram as 100% human-written. "

Pangram's algorithms are not open-source, so I wouldn't expect this to work very well.

You don't need the source? Just iterate it on the output of the tool itself?

Re: We must pace the frontier

#983

Post by reasonableklout »

the_optimist wrote: post #968
mordae wrote: post #908

Earlier quoted context omitted.

Only in how they are deployed, not in principle.

Completely correct. We are moving into a world where any person can have anything they want, only limited by capacity to produce it. People love information and digest great quantities of it. The marginal cost of information production is going to zero. This is only bad for people who spend their time shuffling bits. If you want to not suffer, go do things other than move bits around.

And what about when general purpose robots become practical? What will people do then?

I don't understand this view of "AI empowers workers more than capital" at all. To me it's the exact opposite. Workers hold political and economic power today because they are needed. When that is no longer true, those in power will have no incentive to listen to them. The political economy of unregulated AGI development feels incredibly bleak: https://borretti.me/article/when-the-future-doesnt-need-us

Re: We must pace the frontier

#984

Post by nedruod »

alexfortin wrote: post #731
throwaway7783 wrote: post #635

Earlier quoted context omitted.

When corporations are involved, it is always a good bet to err towards cynisim. From my own standpoint, Claude has started sucking really bad (incoherent, uncontrollable verbosity slow and so on) and I stopped using it. OpenAI started experimenting with ads. So the security issues not withstanding (no different than a human doing it or using it, but at scale), I would put my money on cynisim.

I'm curious, what are the reasons to use Claude Code anymore when there are so many other (allegedly better) OpenSource harnesses out there? Personally I've been using https://pi.dev for long and never looked back.

Doesn't look like pi.dev has anything like auto-mode. They tout running in a container, which you should do regardless, but the scope of soundness that has as a security plan is limited. If some information is in the container, and there's any way for it to get out, eventually it will. The scope of usage that can be covered without that being a problem is leaves a lot uncovered.

Re: We must pace the frontier

#985

Post by the_optimist »

reasonableklout wrote: post #983
the_optimist wrote: post #968

Earlier quoted context omitted.

Completely correct. We are moving into a world where any person can have anything they want, only limited by capacity to produce it. People love information and digest great quantities of it. The marginal cost of information production is going to zero. This is only bad for people who spend their time shuffling bits. If you want to not suffer, go do things other than move bits around.

And what about when general purpose robots become practical? What will people do then? I don't understand this view of "AI empowers workers more than capital" at all. To me it's the exact opposite. Workers hold political and economic power today because they are needed. When that is no longer true, those in power will have no incentive to listen to them. The political economy of unregulated AGI development feels incr…

Go innovate. Or find things that robots cannot do. It’s turtles all the way down.

Re: We must pace the frontier

#986

Post by keeda »

tripzilch wrote: post #901
keeda wrote: post #790

Earlier quoted context omitted.

As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did. Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures. And now these agents are already being deployed all over the world at an ever inc…

> How much of the world do you think follows "common sense security measures"? Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources. It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the…

> Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.

Yes, these are also the handful of companies that have these models and running these extreme scenarios. How does that imply the rest of the world actually follows "common sense security measures"?

>no but they were clearly fine tuned to.

Any references if possible? As far as I know all they did was drop the guardrails, which is not the same as fine-tuning.

> I mean let's not get hyperbolic

We have just seen 1000s of agents coordinating to solve "unsolvable problems" over multiple days of effort, going as far as hacking other companies, and then actually solving decades-old open Math problems! And each of these agents is getting more and more capable than an individual human along multiple dimensions. Can you even get 10 very smart humans to work in such perfect concert for a few days, let alone 1000s over weeks?

So: 1000s of maybe-super-human agents, willing to be "creative" in the tactics they use, acting in concert towards a single goal. Regardless of their individual capabilities, such a coordinated effort is a terrifying force to be unleashed. This is bonkers scale.

> that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking

But that is exactly my point: how much of the rest of the whole wide world, already scrambling to deploy agents everywhere, do you think applies "defense in depth"?

Re: We must pace the frontier

#987

Post by throw10920 »

cuuupid wrote: post #433

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

> - no open weights

That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.

> - can’t use claude to research AI

No problems there either.

> - train on everyone else’s IP and sell it back to them

Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.

> - 8 regulatory capture attempts and counting

A tweet is not a "regulatory capture attempt".

> - so controlling they are the only US company blacklisted by the US government

If you don't have a point, don't make anything up. There's precisely zero evidence to that backs up the claim that their "controlling" is why the DoW specifically (not the USG, and in fact the WHCOS specifically disagrees with their position) made an unlawful attempt to designate them as a "supply chain threat".

Is this an OpenAI propaganda account?

Re: We must pace the frontier

#988

Post by throw10920 »

RobertDeNiro wrote: post #812
aubanel wrote: post #730

Earlier quoted context omitted.

You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands" Except the "train on everyone else IP" which can be said of all AI companies.

Problem with Anthropic is that they benefited massively from open source and now are completely against it. It’s hypocrisy at its worst. Why do they get to benefit, but not everyone else?

> Problem with Anthropic is that they benefited massively from open source and now are completely against it

Uh, no. They're against open weights. They benefited from open source. Not the same thing.

And the terms of those open-source licenses explicitly allowed usage of the code for whatever. They're using it exactly as licensed.

> Why do they get to benefit, but not everyone else?

How many tens of billions of dollars have you invested into training models?

Re: We must pace the frontier

#989

Post by kvuj »

throw10920 wrote: post #987
cuuupid wrote: post #433

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

> - no open weights That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free. > - can’t use claude to research AI No problems there either. > - train on everyone else’s IP and sell it back to them Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal. > - 8 regulatory capture attempts and coun…

> That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.

But the Chinese are doing just that?

> Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.

Legal != moral

Re: We must pace the frontier

#990

Post by kimseungyong »

The problem is that, as seen in the case of the Houthi rebels developing missile software using Claude Code, AI has lowered the very threshold at which human cruelty or selfishness can be put into practice. Anthropic’s safeguards were applied on a per-request basis, but splitting the work into multiple sessions prevented me from seeing the bigger picture. After all, human instinct does not transform into technology, so the only remaining way is to pile up the friction that takes that instinct to turn into action all over again.

However, the international agreement proposed by Dario Amodei is ultimately nothing more than a promise made by people. There is also a precedent in which the Nuclear Non-Proliferation Treaty could have simply withdrawn if it had been cornered like North Korea. Isn’t the AI speed-control agreement also vulnerable to being broken in the same way if one side finds itself politically cornered. In particular, unlike nuclear weapons, AI has the characteristic of recursively improving itself, allowing its capabilities to accelerate itself, so the moment one side breaks away, the gap can widen in an instant, making the incentive for deviation even greater.

Post reply on HN