If AI scaling is to be shut down, let it be for a coherent reason

Obviously as I got older and went on to learn about AI and neural nets in university my opinions on this subject matured. Instead of worrying about killer robots my thoughts became more nuanced, however what didn't change was my fear – if anything this increased from a silly childhood fear of a terminator-like robot, to unleashing an uncontrollable super AI which is indifferent to human life.

Finding Eliezer's work sometime in the early 2010s was almost a therapeutic experience for me. Finally someone understood and was talking about the kind of concerns I had and was accused of being a lunatic for thinking.

My primary concerns with AI basically boil down to two things:

1. Politics is largely a power struggle. Democracies give power to the collective, but this power is cemented by the reality that political leaders and institutions could not function without taxation and labourers.

The reason the governments can't just do things that are deeply unpopular and suppress any public revolts with armed responses is because eventually you will either have to kill everyone or everyone will just stop working in protest. Either way the entire system will collapse.

AI being able to create wealth without human labours and industrialised weaponry fundamentally removes the power dynamic needed to support democratic societies – you cannot withhold labour and you cannot overpower the state with force.

At the same time it would also make humans a waste of resources to those in power, as the unemployed masses are basically just leeches on the resources and space of the state – resources which could otherwise be hoarded by those in power.

If super AGI systems ever exist, the state will be all-powerful while having even less incentive to care about your opinions as it does a rat's.

2. Super-intelligence is fundamentally uncontrollable. I can't even be bothered to argue this point it's that self evident. If you disagree you either don't understand how modern AI systems work, or you're not thinking about the control problem deeply enough.

But the argument here is simple – all other technologies with destructive capabilities rely on human decision making. No gun is ever shot, no war is ever started, no nuke is ever dropped without a human decision maker. Super-AI removes this control from us and requires a kind of hopeium that AI will just be nice to us and care about improving our standards of living. And for those who want to argue "you can just program it to be good" – no you can't, for the most part we have no clear understanding of how advanced AI systems of today internally operate.

The alignment conversation is fundamentally pointless in regards to a super-intelligence because you can not reason with any level of certainty about the alignment of an intelligence far superior to your own.

Instead we should assume unalignment by default because even if you assume we can somehow create an aligned super-intelligence the likelihood that eventually an AI system would be created which is unaligned is practically 100% – this is true even if you assume humans will never intend to create it. Why? Because in a world where we can all create a super-intelligent aligned AIs, you also have near-infinite opportunities for someone to create an unaligned one.

And here's another statement you should assume by default – the only world in which nukes aren't going to be used destructively again is a world in which humans or nukes no longer exist. The same will be true for AI systems, but this time the barrier for creating destructive AI is likely going to be orders of magnitude lower than that of nukes, so unlike nuclear holocaust we don't even have probabilities on our side here.

--------

Look, we're doomed. Progress isn't going to stop and AGI isn't going to be the fairytale people are crossing their fingers for. I don't quite know why no one really seems to have clocked onto this yet, but this will soon become obvious, by which time it will be too late.

Those who think AI is going to improve their productivity are suffering from some kind of delusion in which they believe their ability to type text in a website and copy and paste its output is a marketable skill. It's not, and you're going to be unemployed by tools like GPT in the near-future even without the existential risk posed by super-AGI.

The letter should instead simply have clarified the risks and outlined a sensible, actionable plan to promote responsible development alongside the inevitable march of progress.