Everyone should slow down AI development except for me • Hacker News Board

Re: Everyone should slow down AI development except for me

#191

Post by defrost » Sun, Sep 13, 2026, 6:05 AM UTC

metalcrow wrote: post #174

defrost wrote: post #160

Earlier quoted context omitted.

> We don't have pocket nukes, What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ? > and there's very little risk of random countries acquiring their own due to the amount of work involved. And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US com…

North Korea got it with help from China, Israel got it with help from France. They didn't do it on their own.

And Pakistan with help from the Saudi's, and the US with help from an Australian and the Brits and Canadians

Do you have a point here?

Re: Everyone should slow down AI development except for me

#192

Post by Epa095 » Sun, Sep 13, 2026, 6:09 AM UTC

nradov wrote: post #125

bpodgursky wrote: post #124

Earlier quoted context omitted.

? I said IF the US and China agree to joint monitoring, THEN we could verify how much compute existed and what it was being used for. YES, China will catch up in chip production, but if each side allowed the other to track GPU production and deliveries, on the ground, it would be very hard for either player to have secret LLM training facilities capable of training frontier models.

Nope. Even if China agrees they'll hide their real activities. You're so naive.

If the Chinese loves their children too.

Re: Everyone should slow down AI development except for me

#193

Post by fc417fc802 » Sun, Sep 13, 2026, 6:10 AM UTC

metalcrow wrote: post #173

fc417fc802 wrote: post #162

Earlier quoted context omitted.

Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program. In contrast, rewind to the early 1800s and there is zero hope of a ban on the R&D of small arms being effective in the long run. The only thing it might maybe ensure is that no legitimate actors that fall under your jurisdiction are involv…

> Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program. Isn't that true of AI as well? Data centers are very big and very expensive. > Bit of a tangent but we do, actually Sure but not like the movies, these don't destroy cities. But the metaphor isn't accurate anyway: nukes don't get worse.…

You don't need a datacenter to train a 30B model. Further, the rapid trend of increasing efficiency and decreasing model size for a given level of capabilities means that what is possible to do without a datacenter will continue to increase. Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there. Ergo I expect such an approach to regulation would prove entirely self defeating.

Remember, as I mentioned earlier the human brain only consumes on the order of 20 watts and fits in a handbag. Would you have us destroy all chip fabs? Ban all biomedical and genetic research? How far are you imagining this butlerian jihad would go?

Re: Everyone should slow down AI development except for me

#194

Post by anon373839 » Sun, Sep 13, 2026, 6:12 AM UTC

rtpg wrote: post #183

anon373839 wrote: post #172

Earlier quoted context omitted.

> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy. You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.

DGX Spark is a biiiiit costly but neat to hear! Do you know what kinda throughput you’re getting on that kinda setup? (I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)

It is costly, especially right now. I don’t think you can make a case for it on cost savings!

The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.

I think these numbers may improve because the model is pretty new and optimizations aren’t done.

Re: Everyone should slow down AI development except for me

#195

Post by Paracompact » Sun, Sep 13, 2026, 6:15 AM UTC

metalcrow wrote: post #155

Paracompact wrote: post #138

Earlier quoted context omitted.

It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.

Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.

Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.

Re: Everyone should slow down AI development except for me

#196

Post by ricksunny » Sun, Sep 13, 2026, 6:19 AM UTC

dorolow wrote: post #46

nradov wrote: post #36

Earlier quoted context omitted.

Yikes. Hopefully you never gain any power or influence. The last thing we need is more economic central planning. I sincerely hope that you fail.

Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.

> Do you think things like the Manhattan Project were a mistake

Yes. At best now the mother of all cautionary tales.

Re: Everyone should slow down AI development except for me

#197

Post by necovek » Sun, Sep 13, 2026, 6:21 AM UTC

ben_w wrote: post #177

Buttons840 wrote: post #104

Earlier quoted context omitted.

All the worst outcomes involve taking this technology away from the public.

The current state of alignment is that we don't know how to make it only as bad as Pol Pot was made by his parents, teachers, genetics, circumstances, etc. This remains true regardless of if it is or isn't kept away from the public. "Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical gr…

I agree an argument can be made that not all the worst outcomes include restricting public access, but focusing on niche extremist groups (who might also get access to it through government programs by some of them being government employeed) is not a good argument to protect the public.

Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.

Re: Everyone should slow down AI development except for me

#198

Post by owisd » Sun, Sep 13, 2026, 6:21 AM UTC

augment_me wrote: post #133

metalcrow wrote: post #92

A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up g…

Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the wor…

The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector.

The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.

Re: Everyone should slow down AI development except for me

#199

Post by ryandrake » Sun, Sep 13, 2026, 6:27 AM UTC

Marsymars wrote: post #91

syrrim wrote: post #65

Earlier quoted context omitted.

You trust trump more than any of those three? Really?

You can have an arbitrarily low opinion of Trump and still trust in democracy as a form of government over autocracy or oligarchy, which is what the AI labs have.

Exactly. No matter how bad Trump is, millions of American people at least voted / vouched for him. Nobody voted for Elon Musk, Sam Altman, or any other CEO to have as much power as they have. Maybe a handful of shareholders. Totally unelected and unapproved by the public, but run companies that affect most of the public's lives.

Re: Everyone should slow down AI development except for me

#200

Post by ben_w » Sun, Sep 13, 2026, 6:29 AM UTC

dosisking wrote: post #147

Buttons840 wrote: post #104

Earlier quoted context omitted.

All the worst outcomes involve taking this technology away from the public.

Not necessarily, it just guarantees that China will win.

Unclear.

How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)