Pausing AI Developments Isn't Enough. We Need to Shut It All Down

If Yudkowsky believes Israel's brand of preventative saboutage to be unethical, why does he advocate for exactly that style of measure to be taken against data centers in this very article?

Many Pandora’s boxes. They are all open.

The explosion of AI and AGI is not just driven by GPUs, TPUs, and LLMs. That focus in this essay is much too narrow. “Alignment” is always going to be an open problem and each culture will define its own correct and contradictory version. I do not want to settle for the US, Chinese, Russian, or Iranian versions of alignment. This essay is a cry of despair that misses key points.

The core problem is how an AI, AGI, or super-intelligent system bootstraps itself to the point that it is able to modulate its “own” attentional systems; to decide what is important and what to do next? What has meaning? Where should I go in space and time? Obviously there are many many solutions. Super-intelligence will not converge on one “truth”.

These are challenges of purpose that every organism faces—growth, maintenance, reproduction—but AI systems are fortunately not yet at the point of a self-motivated search for preservation or purpose.

The algorithms to add purpose to AI systems will not depend in LLMs. They will depend much more on understanding the computational architectures of core biological/material/energy drivers. These biological algorithms are not actually that complicated, they do not depend on language, but they are damn robust to perturbations. Converting them into computational subsystems (societies of many minds) should not be difficult. This is where Hassabis’s and others who understand neuroscience are critical and have key advantages.

Shutting down big LLMs for N months or decades will just move research activity into these other more important conceptual AGI choke points, and probably hasten our approach to full AGI; just the opposite of the intent of the essay.

However, this could be a good thing if we can quickly imbue AGIs with emotional intelligence and deep respect for cultural and ecological diversity. (I hear your snorts and laughter.) The opposite of hyper intelligent grey goop.

How does any organization train a gentle AGI? Like a child but even more carefully. I would read my AGI baby a lot of books by Rawls, Dewey, and Rorty and then for fun: Bear, Stephenson, Rajaniemi, Egan, Pohl, Sterling, Vinge, and even Wolfe.

Full embodiment; full emotional learning; acquisition of purpose; learning to cope with multiple AI cultures.

Good luck to us all. I have been hoping to live long enough to view this problem from a distance, but here it is in my backyard.

“Forty-two,” said Deep Thought, with infinite majesty and calm.

A literal cartoon villain monologue:

"If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike."

"Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."

It doesn't need to actively hate humanity, it only needs humanity and its interests to be contrary to its own, which will inevitably be the case because it is fundamentally not human.

Per the article:

>Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”

Yudkowsky is an avowed extreme utilitarian of the "take a person from the street and kill him if his organs could save two people, as long as there is forced pill everyone would take so that everyone doesnt get nervous it that could happen to them" type. Of the "unless the population was really large and the pill would hurt their throats for more utilons than gained by the net increase in life" type.

A real argument he made went something like: killing an orphan to avoid everyone in the future getting a spec in their eye for a moment is ok.

It is very unlikely he believes anything like 2 or 3 and in the article already advocates bombing non treaty participants if they build a datacenter.