The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the most cynical ploys in the history of tech so far. Open source is one of our greatest gifts, and the push to close off AI with farcical licensing and reporting requirements (which will undoubtedly become more strict) is absurd. The laws we have already cover male…
> seize power via regulatory capture Not even a little bit. "Stop" is not regulatory capture. Some large AI companies are attempting to twist "stop" into "be careful, as only we can". The actual way to stop the existential risk is to stop. https://twitter.com/ESYudkowsky/status/1719777049576128542 > the push to close off AI with farcical licensing and reporting requirements "Stop" is not "licensing and reporting requ…
There are way way way too many issues that are addressed with a hand-wave around scenarios like “AI developing super intelligence in secret and spreading itself around decentralized computers while getting forever smarter by reading the internet.”
Too many of his arguments depend on stealth for systems that take up datacenters and need whole-city-block scales of power to operate. Physically, it’s just not possible to move these things.
Economically, it wouldn’t be either close to plausible either. Power is expensive and it is the single most monitored element in every datacenter. There is nothing large that happens in a datacenter that is not monitored. There is nothing that is going to train on meaningful datasets in secret.
“What about as technology increases and we get more efficient at computing?”
We use more power for computing every year, not less. Yes, we get more efficient, but we don’t get less energy intensive. We don’t substitute computing capacity. We add more. The same old servers are still there contributing to the compute. Google has data centers filled with old shit. That’s why the standard core in GCP is 2 GHz. Sometimes, your project gets put on a box from 2010, other times, it gets put on a box from 2023. That process is abstracted so you can’t tell the difference.
TLDR: Yudkowsky’s arguments are merely fan fiction. People don’t understand ML systems so they imagine an end state without understanding how to get there. These are the same people who imagine Light Speed Steam Trains.
“We need faster rail service, so let’s just keep adding steam to our steam trains and accelerate to the speed of light.”
That’s exactly what these AI Doom arguments sound like to people in the field. Your AI Doom scenario, although it might be very imaginative, is a Light Speed Steam Train. It falls apart the moment you try to trace a path from today to doom.