Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We must pace the frontier

darioamodei.com

Post reply on HN
Print view View original

881–890 of 951 posts

Re: We must pace the frontier

#881

Post by eueej »

panarky wrote: post #869
FusionX wrote: post #782

Earlier quoted context omitted.

We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulat…

>> I don’t understand all the comments assuming that RSI is the real threat here > leaked private data, loss of life This smells like more of a money move than a safety move. Amodei is proposing to form a cartel of American frontier labs. They all agree to shift compute away from cash-burning research and training toward cash-generating inference. Then tacitly agree not to compete on price. They'll install independen…

Yup it’s tacit collusion.

I would’ve thought If they had AGI they could come up with a 10d chess move.

Nope.

Imagine how stupid you gotta be to believe their nonsense.

The reality is it doesn’t matter what they do. China is always one step behind and will continue its open product strategy approach.

Re: We must pace the frontier

#882

Post by spopejoy »

ls612 wrote: post #707
lelanthran wrote: post #657

Earlier quoted context omitted.

> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work. An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself. A few programs to break in,…

The GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable. More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and…

It is really mysterious how awestruck folks are at the HF attack. I mean just watch one modern agent (qwen 3.8, deepseek 4, glm 5.3) rip apart a coding problem and nearly destroy your computer in the process, it's a wonder it took "months" and "millions" in the first place.

OpenAI has too much money, a common post-growth-stage issue that leads to pursuing a million stupid things with no clear plan. Like running a bunch of agents for months without any idea how to keep track of progress.

Re: We must pace the frontier

#883

Post by narnarpapadaddy »

PantaloonFlames wrote: post #420
narnarpapadaddy wrote: post #208

Earlier quoted context omitted.

I think this is the correct take. And until we have an AI Hiroshima it’ll be difficult to get the international community to work together. It’ll take rogue AI (or AI-powered group) disabling a significant world power before everyone comes to the table. Otherwise, it just looks like MAD and the equilibrium holding to the powers that be.

It wasn’t the shock of Little Boy that ended the war. That was just a final chapter of a 10+ year journey of horror, despair, and destruction, and the impact of Little Boy cannot be considered outside of that journey. Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight. If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. P…

No disagreement here; I felt countless smaller problems leading up to the final blow was implied.

Re: We must pace the frontier

#885

Post by hex4def6 »

Loquebantur wrote: post #552
hex4def6 wrote: post #534

Earlier quoted context omitted.

The greater the competitive advantage, the greater the incentive to "cheat" the rules. I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper. The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stock…

People don't "play by the rules" for uniform reasons: some have rational insight, others can be enticed and still others need to be shown averse consequences for breaking them. Rules aren't beneficial only when you hit 100% conformity, the mere possibility of them getting broken is no rational reason to abandon rules altogether. "Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerfu…

> ...the mere possibility of them getting broken is no rational reason to abandon rules altogether.

Of course I agree with that. The problem is the enforceability / detectability of this on a global scale hovers around zero, when we're talking literally trillions of dollars on the line.

You would literally have to have every data center account for every cycle of compute it generates.

Again, the nearest comparison feels like bioweapon and nuclear bans. But the difference is the incentives for building those feel significantly weaker, so treaties that reduce and ban their creation are pretty easy to get consensus on, broadly speaking. By comparison, this is something that might provide a 10x competitive advantage to your country's economic, intellectual, and military might.

I think the only way we would get broad global consensus on this is for something Really Bad to happen, so that it moves it out of the fuzzy "sci-fi theoretical" into concrete existance.

Re: We must pace the frontier

#886

Post by taneq »

pvab3 wrote: post #780
MisterTea wrote: post #713

Earlier quoted context omitted.

I feel like all the closed model providers are milking it as they likely know open models on local hardware will one day eat their lunch. We all know it's not a matter of if but when. The company goes bankrupt, the hardware and property sold off, banks holding the bag.

The only way out is to develop a model vastly more powerful and capable that we have now. The market believes theres a good chance of that, although I've never understood why its truly winner-take-all

Because in the event that someone does build a strongly superhuman AI, no-one else will get a chance?

Re: We must pace the frontier

#887

Post by mapontosevenths »

adsharma wrote: post #873
mapontosevenths wrote: post #855

Earlier quoted context omitted.

> We need to insist on building tech that's explainable by design. You realize this means insisting on terrible tech that humans can understand right? It essentially caps human progress at some point about 4 years ago. If you are old and happy with the way things are this might sound like a good idea. It does not to me.

Oh you want tech that helps discover new science instead of parroting existing wisdom? There is little evidence that the RSI we are discussing is capable of inventing the theory of relativity (or the more advanced equivalent). All we have seen is pattern matching in a much larger space than humans can, with some human provided verification tech. I would argue that human-AI collaboration with explainable tech has a be…

You are assuming that humans are capable of understanding everything. They are not.

Human comprehension sets a ceiling on progress.

It reminds me of those schools that can only teach as quickly as the dumbest kid in the room can follow. We don't want that for our entire species.

Re: We must pace the frontier

#888

Post by qnleigh »

boshalfoshal wrote: post #807
qnleigh wrote: post #318

Earlier quoted context omitted.

> OpenAI / Anthropic models have largely stopped advancing I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.

Yes these guys are completely delusional. 2 years ago a model could barely solve the AMC, 1 year ago it got IMO gold, and this year models have solved multiple millenium problems. Even 1 year ago ai code was just unusable (claude code only became available 1.5 years ago!) and now basically everyone I know from independent shops all the way to faang and anthropic/openai themselves exclusively use some AI agent to code…

Waitwaitwait multiple millennium problems? What was the other one???

Increasing a bound on the fraction of Reimann zeros on the critical line doesn't count; even Anthropic says they don't think this line of work will lead to a solution.

Re: We must pace the frontier

#889

Post by adsharma »

mapontosevenths wrote: post #887
adsharma wrote: post #873

Earlier quoted context omitted.

Oh you want tech that helps discover new science instead of parroting existing wisdom? There is little evidence that the RSI we are discussing is capable of inventing the theory of relativity (or the more advanced equivalent). All we have seen is pattern matching in a much larger space than humans can, with some human provided verification tech. I would argue that human-AI collaboration with explainable tech has a be…

You are assuming that humans are capable of understanding everything. They are not. Human comprehension sets a ceiling on progress. It reminds me of those schools that can only teach as quickly as the dumbest kid in the room can follow. We don't want that for our entire species.

I'm sympathetic to this argument.

All I'm saying is: if you have a choice between two systems with equal power of discovery and one is more understandable than the other, we choose the more understandable one.

Limits of Human comprehension and quest for power are two different motivations that could lead to black box systems that are marketed as semi-explainable.

We need to verify that human comprehension is actually limiting progress before allowing such things and even when we do, do it responsibly on an explainable foundation.

Re: We must pace the frontier

#890

Post by cruffle_duffle »

braydenm wrote: post #270

Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies…

Sorry but… uh… bro. I always wondered if opus 5 was a regression or intentional. I’m seriously thinking it’s actually derived from how people at Anthropic talk.

The last people on earth who should be regulating this are governments and tech oligarchs. I find that vastly more scary (and plausible) whatever unintelligible nonsense opus and fable spew out these days. Seriously. The way those models talk and behave do more to show the limits of AI than anything else.

Anyway. Touch grass.

Post reply on HN