We must pace the frontier • Hacker News Board

Re: We must pace the frontier

#621

Post by api » Sat, Sep 12, 2026, 9:27 PM UTC

zozbot234 wrote: post #577

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…

My personal belief, or at least strong hypothesis, is that this kind of recursive self improvement without real world embodied feedback of some kind is impossible.

I think it violates a conservation law. RSI “foom” to superintelligence is an informatic analog to an infinite energy or perpetual motion machine.

To get smarter you must try to solve real problems in the universe and then do some kind of meta learning (natural selection or some other method of refining the intelligence architecture based on an error signal) to iteratively improve your ability to solve real problems. The error signal is outcome measured against a goal function, which for life is survival (probably reducible to genetic fitness and emergent higher order unit fitness from that).

What’s really happening here is learning. To learn, you must have input. You must have training data.

What is the goal function for RSI? Where does the information come from? How do you know if your recursive modifications are making you smarter or just overfitting you to your own idea of smartness?

I predict the latter. RSI will show transient improvement as the current local maximum is optimized and then spiral off into overfitting.

Re: We must pace the frontier

#622

Post by ozozozd » Sat, Sep 12, 2026, 9:27 PM UTC

> In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”

This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.

If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?

3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.

If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.

Re: We must pace the frontier

#623

Post by pessimizer » Sat, Sep 12, 2026, 9:27 PM UTC

aswegs8 wrote: post #601

hgoel wrote: post #586

Earlier quoted context omitted.

The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the inc…

There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.

There's nothing stopping anyone from doing it, even without AI. People have proved entirely capable of doing a lot more hacking than happened here.

Re: We must pace the frontier

#624

Post by ethagnawl » Sat, Sep 12, 2026, 9:27 PM UTC

> I believe that AI could cure most major diseases in the next 5–10 years

This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)

Re: We must pace the frontier

#625

Post by jbellis » Sat, Sep 12, 2026, 9:28 PM UTC

nickysielicki wrote: post #157

angusturner wrote: post #115

Earlier quoted context omitted.

This seems highly optimistic to me. What's to say current AI won't be highly power-concentrating by default? The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class. On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers. No…

> The best models are owned by a few companies In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.

This is only correct if you ignore subscriptions. I ran the numbers on what you get for your subscriptions last week: https://blog.brokk.ai/a-coding-subscription-tier-list/

Re: We must pace the frontier

#626

Post by anon84873628 » Sat, Sep 12, 2026, 9:28 PM UTC

causal wrote: post #536

anon84873628 wrote: post #43

Earlier quoted context omitted.

Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together? For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.

I take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.

I've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question.

The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.

Re: We must pace the frontier

#627

Post by hgoel » Sat, Sep 12, 2026, 9:29 PM UTC

aswegs8 wrote: post #592

zozbot234 wrote: post #577

Earlier quoted context omitted.

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…

Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?

In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc.

AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.

Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.

Re: We must pace the frontier

#628

Post by stevenhuang » Sat, Sep 12, 2026, 9:29 PM UTC

zozbot234 wrote: post #577

RGS1811 wrote: post #521

I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope t…

It takes quite a lack of foresight to think RSI is completely speculative when it's already been demonstrated how capable agents are at long horizon tasks given suitable harness and unambiguous success criteria. It's hardly a leap to give LLM the goal of improving itself on benchmarks and let it conduct it's own experiments and spin up training runs completely unsupervised.

It's strange you believe this can't happen when a weaker form of it is already happening. And to be so certain RSI can't happen when there really is no technical basis why it can't.

Re: We must pace the frontier

#629

Post by matheusmoreira » Sat, Sep 12, 2026, 9:29 PM UTC

busymom0 wrote: post #585

cuuupid wrote: post #457

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

> can’t use claude to research AI What's this about? Where's this rule?

In the system cards. Anthropic will literally make Claude sabotage you silently instead of downgrading you to Opus if you try to use Fable for AI research.

Re: We must pace the frontier

#630

Post by 8note » Sat, Sep 12, 2026, 9:31 PM UTC

Aurornis wrote: post #442

HarHarVeryFunny wrote: post #425

Earlier quoted context omitted.

> Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules. Sure - look at what is being done today with H1B visas and tariffs. What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into? Just as with H1B visas, you restri…

> Sure - look at what is being done today with H1B visas and tariffs. I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy. > What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into? This is the definition of a false dichotomy, and you also ignored (or missed) the point…

free market doctrine is an axiom to your second order effects. its so embedded into all your assumptions that you havent even considered it