Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

751–760 of 946 posts

Re: We must pace the frontier

#751

Post by nwienert »

matheusmoreira wrote: post #659
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets. If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.

Anthropic gives you much more compute with their $200 plan, inclusive of resets, and this has been true for a very long time.

There was only a brief window of time that the opposite was true.

Re: We must pace the frontier

#752

Post by pants2 »

HAL3000 wrote: post #620
zozbot234 wrote: post #577

Earlier quoted context omitted.

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…

Yeah, yesterday's talk[1] goes into detail on this, showing how no one really knows how to tackle it because LLMs don't know how to create their own novel objectives. It's also interesting how many diminishing returns they hit now and how many low hanging fruits are already harvested, it seems like we are approaching the flattening part of the S curve, where further gains become harder to achieve. 1. https://www.yout…

Diminishing returns is extremely hard for me to believe given how fast model releases are going. Six months ago we were on GPT-5.3, and Astra blows it out of the water in every regard. How many times have commentators claimed we're hitting a wall? I don't see any wall.

Re: We must pace the frontier

#754

Post by sega_sai »

While some of the text sounds sensible to me, i.e. some sort of external oversight + reporting of incidents, much of the rest seems focusing on limiting China and their open models as they are clearly approaching the abilities of Anthropic's models.

Re: We must pace the frontier

#755

Post by hkt »

Aurornis wrote: post #445
Loquebantur wrote: post #434

Earlier quoted context omitted.

You start from incorrect premises by presupposing agreed-upon rules would never work anyway. "Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to. Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.

> You start from incorrect premises by presupposing agreed-upon rules would never work anyway. The rules would never work on a global economy. If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy. If you think it’s that easy to get all competitors in a global economy to agree to a set…

Nuclear disarmament is governed by treaty, which is largely adhered to.

An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.

Re: We must pace the frontier

#756

Post by pants2 »

hgoel wrote: post #627
aswegs8 wrote: post #592

Earlier quoted context omitted.

Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?

In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc. AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the…

We don't know exactly what the limits of AI improvement on our current infrastructure are, though. If the human brain is 20W, and a datacenter is 1GW, then maybe that datacenter can be 50 million times smarter than a human. If that's not already a risk to humankind I don't know what is.

Re: We must pace the frontier

#757

Post by solenoid0937 »

icantevenhold wrote: post #744
afthonos wrote: post #738

Earlier quoted context omitted.

Worse than extinction?

There are worse fates than extinction, for example living forever, for I have no mouth and I must scream

I dunno, being a Culture Mind sounds pretty damn good. Or even just a death-optional citizen in the Culture.

Re: We must pace the frontier

#758

Post by AgentME »

zozbot234 wrote: post #577
RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…

Today we prompt software developers to "think about ways to improve AI's architecture" and it results in AI getting better. AI over the last year has made very rapid gains in filling the role of a software developer.

Re: We must pace the frontier

#759

Post by matheusmoreira »

nwienert wrote: post #751
matheusmoreira wrote: post #659

Earlier quoted context omitted.

I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets. If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.

Anthropic gives you much more compute with their $200 plan, inclusive of resets, and this has been true for a very long time. There was only a brief window of time that the opposite was true.

> Anthropic gives you much more compute

That does not match my experience. I switched away from Anthropic to OpenAI roughly a month ago, and it's almost comical how much more usage I'm getting out of this subscription.

I migrated from Anthropic's 5x plan to OpenAI's 5x plan, and eventually upgraded to 20x after I was able to statistically verify that OpenAI plans were almost exact multipliers of the Plus plan, exactly as advertised. Meanwhile, Anthropic has gotten caught playing "20x referred to the five hour limit" word games with their customers.

Re: We must pace the frontier

#760

Post by api »

throwup238 wrote: post #705
api wrote: post #621

Earlier quoted context omitted.

My personal belief, or at least strong hypothesis, is that this kind of recursive self improvement without real world embodied feedback of some kind is impossible. I think it violates a conservation law. RSI “foom” to superintelligence is an informatic analog to an infinite energy or perpetual motion machine. To get smarter you must try to solve real problems in the universe and then do some kind of meta learning (na…

I also strongly hold this belief largely due to Moravec’s paradox, which is kind of approaching this issue from the side. Sort of like large language models work on top of what our language has encoded in our massive training datasets, I think biological intelligence is built on top of the parts of the brain that encode the real physical world. These parts grow/train from embodied experimentation and instinct early o…

I don't think Moravec's paradox is the same, and you could argue that one no longer holds -- though I'm not sure. You could also argue that Moravec's paradox still holds but that we now have such powerful computers and huge models that we have been able to brute force our way to the capabilities it talks about. It takes many many orders of magnitude more compute power to do things like spatial location, language processing, etc. than it does to do more closed-form things like chess... we just actually have that compute power now.
Post reply on HN