Humanity isn't ready for the coming intelligence explosion

Do you have an example of a prediction they’ve made in the last 3 years that was drastically wrong? This is not my impression.

Do you have a source on aggregate predictions of experts being drastically wrong?

I.e. the AI 2027 guys were memed for being AI lunatics on a lot here and they have been pretty on the money in terms of pace of progress accelerating/ gov moving towards nationalization/ coding agents

I wanted this to be thought provoking, but for a physicist to open with, "the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million", just seems sad. It is a phrase without meaning.

   Is it per plant? there aren't a million.
   Is it per year? notably have been at least 2 major ones (arguably 7).

A meltdown (or loss of containment) just isn't that bad, if it doesn't affect ground water or lead to atmospheric fallout. We're turning 3 Mile Island back on to power an AI datacenter! The AI super-intelligence apocalypse envisioned (ignoring the likelihood) is inherently global.

That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.

Honestly, this article has too many words which don't mean anything.

'AI Experts', 'superintelligence', and hand-waving doom scenarios.

As an example-

> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.

I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.

Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.

Ultimately, what counts is what we do with it.