Live data from Hacker News

We must pace the frontier

darioamodei.com

721–730 of 951 posts

Re: We must pace the frontier

#721
post #391

Earlier quoted context omitted.

Military action against whom? the US can’t even handle Iran let alone China.

America doesn't want to pay the price of handling Iran. There's a difference. Clearly the US could conquer Iran but it requires a land war with mass casualties, not just sporadic bombing. Obviously China is in an entirely different league.

"I can afford ferrari. I just don't want to pay the price."

Re: We must pace the frontier

#722

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolisti…

> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...

It's all lies as usual.

This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.

We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".

Re: We must pace the frontier

#723

Earlier quoted context omitted.

If I set my walet beside me and you swipe in walking by, you have still committed theft. Victim blaming isn't legally acceptable

Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences. By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero. https://www.bitsaboutmoney.com/archive/optima…

> User just disputes the charge and it kind of disappears. The banking system just absorbs it,

No it doesn’t.

> It's all priced in.

So you admit awareness that fraud loss doesn’t kind of disappear.

We all pay for it, either via higher merchant fees or higher interest rates, sometimes both, on card purchases.

Re: We must pace the frontier

#724

Earlier quoted context omitted.

These do not seem to be mutually exclusive. Should we not do both?

Do you really think we can really get every country to truly pace the frontier? Pretty sure China won't give a f until they catch up Anthropic and OpenAI. It is an arm race. We had nukes for like 70 years and still haven't figured out how to make every single country follow those nuclear treaties, with an increasingly non-interventionist US I don't think we can get every single country to the table and agree to a pau…

> Do you really think we can really get every country to truly pace the frontier?

Why not? In 20-th century half the world was socialist. And the idea of moratorium on improving AI way easier to sell then socialism.

Re: We must pace the frontier

#725
post #707

Earlier quoted context omitted.

> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work. An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself. A few programs to break in,…

The GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable. More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and…

I agree with the idea that it's not probable but I do think it's important to point out-- we only need these elements to run with the bandwidth they have and the token rate because we want to see things in human-scale time. But slow things down to a 1tok/sec doesn't matter to this hypothetical anti-aligned LLM. Time is, after all, relative, and it's not like LLMs give a shit how long something takes. They don't have squishy stupid organs that fail after a certain amount of time, or those pesky glands that emit impatience hormones.

But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.

Re: We must pace the frontier

#726

Earlier quoted context omitted.

> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI…

Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier than it would be without using LLM tooling? Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier. Fable has only been out for thre…

> Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier

Marginally easier? Yes of course, same as how it's now "easier" to write any kind of code because we aren't using punch cards anymore. That still doesn't get you to any kind of unbounded "takeoff" scenario, because diminishing returns are a thing. The "loop" of people trying out new ideas can only shrink so much.

Re: We must pace the frontier

#727

Earlier quoted context omitted.

That’s only one metric. It doesn’t matter how cheap it is, if it continually fails to solve a problem. Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.

Have you actually used the latest Chinese models? Because, in my experience, they're not just cheaper, they're actually better in many cases. In one recent experiment I did a few days ago on a task that I need in production at scale at my company, Qwen 3.8 and GLM 5.3 Flash were not just cheaper, the results were significantly higher quality than GPT 5.6 Sol. Even if China doesn't continue to release this stuff for f…

Yes, I use the messing Chinese models extensively. Like you, I very much appreciate them and sometimes prefer them.

But there is a bar of complexity at which they fail, and repeated invocations typically doesn’t make much progress. This is a small subset of most work, but it still exists. And yes you can help it along, but in those cases I’d typically break out the big guns.

Re: We must pace the frontier

#728
post #536

Earlier quoted context omitted.

I take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.

I've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question. The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.

"Just don't make mistakes" is naive. You have no appreciation for the scale of agents being run right now, finding the rogue agent is a needle in a haystack operation.

Re: We must pace the frontier

#729
post #446

Earlier quoted context omitted.

> OpenAI / Anthropic models have largely stopped advancing I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.

> This year it became common for people to entirely delegate coding to AI This has been the case for around 2 years now, more reliably - a year. We've mostly stayed there since then. Saying that more people started doing it isn't indicative of significant improvement. Some people just started doing it later. I can't speak about math because I haven't used AI for that application, but I know that there hasn't been any…

> This has been the case for around 2 years now, more reliably - a year.

I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.

Re: We must pace the frontier

#730

Earlier quoted context omitted.

What are you trying to say?

I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?

The fluid system products, assemblies, and services company?
Post reply on HN