Who Aligns the Aligners?

The best we can do is have a Bayesian probability distribution about future outcomes.

I don't ever like holding a singular opinion on anything or converging on a single value for things like p(doom), but generally when I talk about things with people I try to represent my views as singular values or binary opinions. So I agree with you that p(doom) as a concept is silly in the same way as you saying you know you won't be dead in 24 hours is silly.

I can't speak for everyone, but I suspect most people who express a p(doom) haven't thought much about it so I agree with you there too. But I do personally put a decent amount of effort modelling out scenarios so I'll just speak to my own p(doom).

My p(doom) is high largely because I believe certain things about how AI capabilities will scale, how human incentives will shape certain outcomes, and our inability to slow progress in a field like AI. Basically:

- It's almost impossible to significantly slow AI progress without a doom-lite scenarios like nuclear war

- AI will scale to ASI in the coming decades

- Corruption is the norm, and democratic systems today are backed largely by the fact the public hold a monopoly over economic power - and all power derived from economic prosperity

- AI will replace 90% of productive human labour in the coming decades and erode the economic value of the humans, resulting over time in political corruption (resource curse)

- ASI unlocks all tech on the tech tree which primarily requires intelligence (which is likely to be the majority of tech)

- There is far more destructive/dangerous tech later on the tech tree than there is good. E.g., for every cancer cure there is 1,000s bioweapons and new torture techniques we can prompt Claude to create

- Even if humanity could agree on a workable definition of alignment it's a much harder problem than advancing capabilities, therefore capabilities will almost certainly progress faster than our ability to align advanced AI

- The field of alignment at it's limit resolves to basically just deciding between a god that follows the commands of a single or small group of humans (dangerous), or a god that has it's own moral considerations and does not listen to humans (dangerous).

So my p(doom) is largely a product of all of the above, and my general inability to find many neutral or good scenarios of ASI while landing on almost endless ways ASI is likely to go wrong regardless of whether you believe terminator-like scenarios. To be honest the only reason my personal p(doom) isn't higher than 99% is that I tend to weight the opinions of others whose opinions I value quite highly, and most people whose opinions I value have a slightly lower p(doom). I've rerated a lot recently because I was hoping some experts would turn out to be right about there being algorithmic limitations to ASI and things like "alignment by default". But I now think I've over weighted those opinions historically and a lot of the better arguments against a high p(doom) are either now dead in the water or have become increasingly hard to hold with any significant probability.

My best argument for against an AI doom scenario this century would be nuclear war, which I personally discard in my AI p(doom) probability. Allowing for other doom scenarios my AI p(doom) is probably closer to 90%.

This might be scarily true. But we end up in the same problem, which is the fear of most people being wiped up.

Does it matter if the AI is doing it on its own or at the wishes of a small group of powerful people? For all intents and purposes they are both the same thing in the end, for most people anyway.

Move. This is already happening, with water shortages forcing people to move. Climate change will (likely) cause famines and many deaths, but the bigger effect will be mass migration.