AI doomism is quickly becoming indistinguishable from an apocalyptic religion

China is behind in semiconductor manufacturing, but they’ll continue to advance-limiting their access to cutting-technologies like ASML’s photolithography machines will slow them down, but given enough time there is no reason they can’t overcome that too. If the point of sanctions is to hold them back and keep them behind, that could continue to work for a long time.

But if you decide to stop progress in semiconductor manufacturing, in order to avert an AI apocalypse - China no longer has to catch up to a moving target, they have to catch up with a stationary one. And as soon as they’ve caught up, now they can overtake. So sanctions don’t work as a way to stop an “AI apocalypse”, unless you convince all the world’s great powers to cooperate in them. And the potential payoff for anyone who chooses to defect is immense-maybe even world conquest-so how do you make sure they all keep the deal, and none wiull try to cheat?

If anything, I think a world with multiple superintelligent AIs with competing allegiances - a pro-Beijing superintelligence, a pro-Washington superintelligence, etc - is likely safer for humanity than a world with only a single superintelligence, or even many superintelligences that all had the same beliefs/attitudes/goals/objectives

I'm more worried about losses of incentives that fund development or the use of it as gatekeeping that will limit entry. Ie, Europe's plans or getty's lawsuit.

Yes, there is a massive amounts of money, incentive, and mindpower to ward them off. But they're also just mostly letting the doomers run free without pushback. I suppose it could work, if it's to ignore the doomers as they'll make themselves look foolish. But theirs a lot of incentives to attack ai research, from "it will take your jobs", to doing it out of pure spite to hurt tech.

Eg. payments to newspaper's is an example of fairly recent line that had money behind it and partially succeeded. It could get stupid too, such as Australia saying linking needed payment.

Most of the main barrier is fear from US/China about letting the other get the upper hand.

Let's not short-circuit our critical thinking by labelling those concerns cult-ish.

Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic.

> I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do.

I agree, but that's actually what I worry about the most.

I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role.

But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes.

I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah.

In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination.

As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well.

> The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence

This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.

> It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible.