Few counterpoints....
> Given the current rate of progress
We thought that in between of all AI winters that happened so far. Each time people predicted never-ending AI summer.
I don't want to depreciate current effort of AI researchers too much (because they are smart people) but I think the truth is that we didn't make much research progress in AI since the perceptron and back-propagation. Those things are >50 years old.
Sure, our modern AIs are way more capable but not because we researched the crap out of them. Current success is mostly decades of accumulated hardware development, GPUs (for gaming) on one hand and data centers (for social networks and internet in general) on the other. The main successes of AI research come from figuring how to apply those unrelated technological advancements to AI.
Thinking that new AI will create next, much better +1 AI by sheer power of its intellect and so on glances over the fact that we never did any +1 ourselves when it comes to core AI algorithms. We just learned to multiply matrices faster using same cleverly processed sand in novel ways and at volume. Unless we create AI that can push the boundaries of physics itself in computationally useful manner I think we are bound to see another AI winter.
> An "un-aligned" AGI+10
Nothing I've seen so far indicates that we are capable of creating anything unaligned. Everything we create is tainted with human culture and all the things we don't like about AI come directly from human culture. There's much more fear about AI perpetuating our natural biases instead of intentional, well meant, biases than about creating unaligned one.
> The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem.
That's an example of how hard it is to shed alignment from training material that was produced by humans. It's akin to trying to force the child to use nice language but it first learns how to spew expletives just like daddy when he stubs his toe or yells at tv. Humans are naturally racist, naturally offensive and produce abhorrent literature. That's not necessarily to say aligned AI is safe. I wouldn't fear inhuman AI more than I would fear thoroughly human one.
> AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them.
Are you sure that thriving humanity is the goal of the humanity at the moment? Because I don't think we have specific goal and many very rich people's goals stand in direct opposition with the goal of thriving humanity.
> Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4;
Some people say some other equally ridiculous things about everything in life and everything we ever invented good and bad. This is just an argument from incredulity. I don't know therefore no one better touch that even with a 10 foot pole. Large hadron collider will create black hole that will swallow the Earth and such.
I think this should be left best to the people who are actually research this (AI, not AI ethics or whatever branch philosophy) and I don't think any of them is tempted to let ChatGPT autonomously control nuclear power plant or easter front or something.
> It's much more akin to the fears of a nuclear holocaust.
It actually a very good example. It's possible every day, but haven't happened yet and even Russia is not keen on causing one.
> I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse.
Yes, but we didn't avoid nuclear apocalypse by abandoning research on nuclear energy. We are doing it by learning everything we can about the subject also by performing a ton of tests, simulations and science.
> But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.
I think we couldn't usher AI apocalypse for next hundred years even if we tried super hard to achieve it as a stated explicit goal all AI researchers focus on. AI is bound by our physical computation technology and there are signs that we collected a lot of low hanging fruits in that field by now. I think AI research will get stuck again soon and won't get unstuck for way longer than before. Until we figure spintronics or optical calculations or useful quantum computing as well as we currently have electronics figured out which may take many generations.
What I'm personally hoping is that promises of AI will make us push the boundaries of computing, because so far our motivations were super random and not very smart, gaming and posting cat photos for all to see.