We must pace the frontier • Hacker News Board

Re: We must pace the frontier

#571

Post by cubic_earth » Sat, Sep 12, 2026, 8:41 PM UTC

stratos123 wrote: post #497

cubic_earth wrote: post #485

Earlier quoted context omitted.

I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists? The other thing is: look at all the fab capacity that is being brought to bear on this. I gues…

> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists? That might be true, sure. > The other thing is: look at all the fab capacity that is being b…

For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.

You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.

We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.

Re: We must pace the frontier

#572

Post by dismalaf » Sat, Sep 12, 2026, 8:42 PM UTC

franticgecko3 wrote: post #419

dismalaf wrote: post #409

This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent). If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If…

That's just so extremely difficult to believe. Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem. In less than a year.

> started one-shotting serious long horizon tasks.

They one-shot tasks for which there's a git clone one-liner, except worse.

My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...

Re: We must pace the frontier

#573

Post by villish » Sat, Sep 12, 2026, 8:43 PM UTC

simianwords wrote: post #315

TheSisb2 wrote: post #311

Earlier quoted context omitted.

> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up. This is obviously untrue… do you use any of them?

This line will keep repeating because it is necessary for the narrative: AI in general is just hype and unprofitable and all these companies are playing marketing tricks before the IPO after which they will cash out and let the economy crash. This is legit what a lot of people think. To continue this narrative, they have to keep up the charade of "things are not improving".

Add in a heaping dash of anti-american sentiment, and you will get the truth behind the pessimistic commentary.

Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.

Re: We must pace the frontier

#574

Post by Chance-Device » Sat, Sep 12, 2026, 8:44 PM UTC

Aurornis wrote: post #397

Chance-Device wrote: post #306

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. Regulate what, exactly? Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules. There’s a theoretical version where all the countries in the world join hands and agree not to compete…

On competition between countries: you’re imagining the world economy as working in the same way pre and post AGI, I doubt it will work the same way at all.

There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fragments into blocks that tolerate AI to differing levels. Unevenly distributed AGI kills globalism the next day.

For the rest of your reply you ignore the “at least” part in that sentence, and also seem to expect a detailed policy proposal. The details can be worked out; there is some reasonable compromise between what activities AI can and cannot be used for, and what level of capability can be deployed where.

More broadly, you seem to believe in a just world fallacy of unregulated capitalism being an inherent good. It isn’t, and regulations exist even in the United States, so this unregulated state doesn’t exist now.

Moreover, regulations and redistribution are stronger elsewhere in the developed world, and there is a strong argument to be made that the lack of these in the US, and the gap between rich and poor that it causes, is responsible for more suffering than having more of these things outside of it.

Re: We must pace the frontier

#575

Post by hgoel » Sat, Sep 12, 2026, 8:46 PM UTC

rickydroll wrote: post #338

Chance-Device wrote: post #306

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…

Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work. Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial effort…

It's all about wanting to preserve the status quo, complete with all the suffering and strife, because change is scary and uncomfortable.

Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.

Re: We must pace the frontier

#576

Post by matheusmoreira » Sat, Sep 12, 2026, 8:47 PM UTC

pizzly wrote: post #570

I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure…

If any of this happens, it's the end of computing as we know it today. Deeply unfortunate...

Re: We must pace the frontier

#577

Post by zozbot234 » Sat, Sep 12, 2026, 8:49 PM UTC

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".

Re: We must pace the frontier

#578

Post by techblueberry » Sat, Sep 12, 2026, 8:50 PM UTC

rickydroll wrote: post #338

Chance-Device wrote: post #306

I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…

Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work. Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial effort…

I feel like the contra answer to “how do you enforce it” is like- imperfectly but we can do stuff.

Like we can just do stuff. We can fine companies, take away licenses. We are capable!

But I also think - I’m not trying to be too negative here because certainly we should be investing in scientists and other discoveries, but early tech - xerox park, googles 20 percent time. They were playing in an extremely immature space.

You could throw 20 ideas at a wall and create a billion dollar business.

We should do what you’re saying but we should also build institutions and maybe also limit the extent to which we’re building Elysium. If we can build AI models that rival human intelligence we can create laws to protect its dignity.

Re: We must pace the frontier

#579

Post by verdverm » Sat, Sep 12, 2026, 8:50 PM UTC

pastel8739 wrote: post #412

verdverm wrote: post #363

Earlier quoted context omitted.

> bound by safety regulation should be fine I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.

yes but the mention of alignment is inside the "pacing within democracies" section

alignment does not have a uniform definition, so "who gets to decide the right answer?"

(1984 has some thoughts on the matter)

Re: We must pace the frontier

#580

Post by anfogoat » Sat, Sep 12, 2026, 8:50 PM UTC

TheSisb2 wrote: post #38

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he…

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.

Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.

> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.

More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.