Ten advances in mathematics and theoretical computer science

There are dozens of escalating, yet fundamentally less destructive means to prevent a data centers from continuing to operate (if that's the goal). Not starting at those more modest interventions is political malpractice and in opposition to your harm reduction narrative.

Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.

Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.

Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.

If you're referring to the "nuke the data centers" line in my bio, that's tongue in cheek. It's an expression of my sentiment towards AI at the extreme.

To be clear, I don't think any country should seriously be considering dropping nukes on data centers. Targeted strikes on data centers operated by rogue private companies or nation states is acceptable to me – assuming pre-warning of intent and appropriate regulatory foreshadowing.

I suspect we agree this is reasonable?

> Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.

> Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.

No, they're not based on unbounded exponential growth curves at all... I'm actually somewhat skeptical of claims that there could exist an intelligence that far exceeds our own. My assumption is really just that AI will be far faster, far more knowledgable, far more persistent and marginally more generally intelligent than the average human. The first three are already true, the forth we're getting to.

We're on an exponential curve right now and I suspect the rate of progress will accelerate some more as a result of RSI. This will likely continue for 2-3 years (if we have that long) before starting to flatten out.

As for my doom prediction, my median concerns are around around the destabilising nature of ASI.

If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation. I can go into more detail here if you wish.

However, I'd still put these destabilisation risks at under 50% in total. My concern more broadly is the majority of ASI outcomes are doom scenarios. It's extremely hard to imagine a world in which you have a magic box that can grant any wish that intelligence can grant, but which we have a functional civilisation. You have to basically assume all potentially dangerous wishes won't be granted and probability the only feasible way to do that is restrict who can make the wishes – but then you have the concentrate of power risk.

I largely agree with the premise "If Anyone Builds It, Everyone Dies", so we must try to avoid building it. I think we can build ASI safely, but we're clearly not on the path right now and I suspect it might take decades (or more) to figure out how we can build ASI with an extremely high probability of a good outcome.

> Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.

I do this. I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.

I don't think we realistically have a chance of diverting from this path at this point. I have made peace with this but it makes me very sad and haunts my dreams almost every night.

I’m not sure if people just aren’t as aware, but the Jacobian conjecture practically was on par with those other great problems.