Earlier quoted context omitted.
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
We must pace the frontier
611–620 of 943 posts
Re: We must pace the frontier
#612Earlier quoted context omitted.
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
Re: We must pace the frontier
#613I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…
Re: We must pace the frontier
#614Earlier quoted context omitted.
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
Re: We must pace the frontier
#615Re: We must pace the frontier
#616Earlier quoted context omitted.
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
Call me when LLMs can get simple things right. Math is just the manipulation of symbols within established frameworks, we should be getting new math out of LLMs daily and we're somehow still not. They can't even do customer service, which is usually handled by 90 IQ people. I'm not impressed that they can find bugs; memory bugs are obvious when they're pointed out to you, and LLMs are entirely made up of examples and the relationships between them.
These companies are about to crash, and they're afraid they haven't reached the point where they'll have to be bailed out. I'm also subscribing to the conspiracy theory that the companies want the government to step in and create AI regulation boards entirely staffed by people at the current US frontier labs, so they can collude to both raise prices, to get government contracts, to make open/Chinese AI illegal, and to make things that were once easy to do without an AI intermediary impossible to do without an AI intermediary. Raising prices and forced purchases are the goal. They're trying to avoid having to compete, because as a business they're garbage.
Matt Stoller characterized their relentless press releasing as something like "my dick is so big that it has to be regulated." It's such an oversell for something that is not showing up as productivity gains, and anybody who has personal experience with knows is incapable of doing more than three things correctly in a row.
Re: We must pace the frontier
#617Earlier quoted context omitted.
On competition between countries: you’re imagining the world economy as working in the same way pre and post AGI, I doubt it will work the same way at all. There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fra…
I don't think this adds up. The AGI blocks will have AGI kill bots and those that don't won't have a choice about what they will or will not allow.
Re: We must pace the frontier
#618Earlier quoted context omitted.
Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?
> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI…
Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier.
Fable has only been out for three months. Astra is even newer. The capability of these models compared to what existed even a year ago, and the effect they are having on the production of new software, is immense.
That's all you need. RSI can happen with what we have now, just by enabling the continuous shrinking of the loop of people trying new ideas and implementing them. It does not require some magical "go make yourself better" prompt against some model that is past some magical tipping point.
Re: We must pace the frontier
#619Earlier quoted context omitted.
On competition between countries: you’re imagining the world economy as working in the same way pre and post AGI, I doubt it will work the same way at all. There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fra…
I don't think this adds up. The AGI blocks will have AGI kill bots and those that don't won't have a choice about what they will or will not allow.
will have nukes.
Doesn't matter how much "intelligence" you have stockpiled when nukes start raining down on you.
Re: We must pace the frontier
#620I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…
It's also interesting how many diminishing returns they hit now and how many low hanging fruits are already harvested, it seems like we are approaching the flattening part of the S curve, where further gains become harder to achieve.