The illusion of AI’s existential risk

Given that the number of people in agriculture declined from 90% to 10% over the past 200 years or so, I'm less worried about disruption like this. People are remarkably retrainable, and just because current jobs no longer exist doesn't mean we won't create new ones out of thin air (like marketing, or Youtubers, or film lighting specialists).

Why wouldn’t you call a human brain “statistical pattern regurgitator” as well, if it comes to this? Matching learned patterns is a big part of how brains work.

I don’t see why a strong AI _needs_ to be trained specifically on reality. Imagine an alien race that sees world in X-Rays and neutrinos. An alien AI researcher claims that a “true intelligences” must train on the same inputs, and all those systems trained on the part of the spectrum between infrared and ultraviolet only fake intelligence.

I don’t see why intelligence has to be analogue in any way. Analogue systems are inherently noisy and contain limited amount of useful information, so you can always faithfully approximate them using digital representations.

If I am not mistaken Deepmind has shown spontaneous emergence of communication around 5 years ago when training agents in multi-player team shooters.

> instead of trying to actually debate the points raised, which are absurd

https://en.m.wikipedia.org/wiki/Appeal_to_the_stone

“Appeal to the stone, also known as argumentum ad lapidem, is a logical fallacy that dismisses an argument as untrue or absurd. The dismissal is made by stating or reiterating that the argument is absurd, without providing further argumentation. This theory is closely tied to proof by assertion due to the lack of evidence behind the statement and its attempt to persuade without providing any evidence.

Appeal to the stone is a logical fallacy.”

One day, someone came up with the Transformer architecture for neural networks. The ideas didn't quite come out of nowhere, but their impact did - I don't think that anyone predicted the degree to which ChatGPT and its ilk are superior to LSTMs and the like. More importantly, nobody predicted the details, the way that the GPTs behave, the way they're willing to invent facts like they're spooling out plausible thoughts without being constrained to observed reality.

From a history-of-technology standpoint, the state of the art for generative AI went from LSTMs to ChatGPT overnight. There was no warning. Nobody could have looked at the Attention Is All You Need paper and predicted how much it would improve the state of the art.

And worst of all, we already know that AIs can be hostile. What else is a public corporation's corporate bureaucracy but a Chinese Room running an AI whose goal is to maximize quarterly dividends? What else is a government but another Chinese Room trying to survive? We talk about corporations being faceless or friendly, we ascribe them wants and needs and behaviors and habits, we give them names, and they suffer natural selection and mutation and reproduce and evolve over time so as to select from those bureaucracies that are most fit for survival. And this class of AI is already responsible for global climate change and the Cold War and so many other things that might already be on the path to push us into extinction.

LLMs are nowhere near as humanlike as a corporation. Corporations are bound by regulations and capitalism and basic human decency. It is absolutely possible to build an LLM whose fundamental goal is to kill people. All you have to do is tell ChatGPT to pretend it's playing a game of global thermonuclear war, and some joker on 4chan would do that in a heartbeat.

So what? Why does this matter?

We have no way to predict when the next revolution in AI capability will occur.

We have no way to predict what the next revolution will be.

We have no way to predict the capabilities of the next generation of generative AIs.

We have no way to predict how the next generation of AIs will "think", insofar as they do.

And because we have no predictions, we have no way to prepare.

So the real question is, how much risk are you willing to tolerate? Are you wiling to take a coin flip on it? A spin of the cylinder? A roll of the dice? Because that's what we're looking at here - the next generation of AIs has no reason to be any less inimical to human existence than the examples we already have, and AI will _only_ get better.

Yes, but at our current scale? At first glance it really seems that our simple economical metrics are a good way to have billions of humans living largely peacefully together – while ruining their environment of course.