> For example, one actual major human tragedy caused by a generative AI model might suffice to push me over the edge. (What would push you over the edge, if you’re not already over?)
Deepfakes have already caused several. Actually, they're more dangerous than the current generative approaches. The first major use case for deepfakes was making convincing looking revenge pornography, as a psychic weapon on people. Dropping deepfake porn on people is a very, very reliable way of getting them to kill themselves[0]. Ignoring that, we also have deepfake-assisted social engineering, which can be scary good if you don't know the specific faults with those kinds of models.
The only pro-social application of deepfake technology was body-swapping actors in popular movies for memes. This was probably not worth the cost.
>we’ll know that it’s safe to scale when (and only when) we understand our AIs so deeply that we can mathematically explain why they won’t do anything bad; and
GPT-3 is arguably Turing-complete[1] and probably has a mesa-optimizer[2] in it. We're able to make it do things vaguely reminiscent of a general intelligence if you squint at it a little and give it the Clever Hans treatment. So I don't think we're ever going to have a GPT-n that's "completed it's morality testing" and is provably safe, for the exact same reason why Apple won't let you emulate Game Boy games on an iPhone. You can't prove the security properties of a Turing-machine or arbitrary code written for it.
I should point out that most AI safety research focuses on agents: AI programs that observe an environment and modify it according to some parameters. GPT is not in and of itself that. However, if we give it the ability to issue commands and see the result (say with ChatGPT plugins), then it becomes an agent, and safety problems become a relevant concern.
The author of the post seems to be unconcerned by the "AI could be worse than nukes" argument. Neither am I, and I think the "six month pause" is kind of silly. However, there are still relevant safety problems being brushed under the rug here.
Also anyone saying the military should bomb GPU farms is daft. They didn't even step in to stop crypto and that was a deliberate attack on central banks.
[0] As far as I'm aware, nobody has killed themselves because of something Stable Diffusion has drawn. Yet.
[1] For the colloquial definition of Turing-complete. Technically speaking it is a linear-bounded automaton because it has a fixed memory size. However, every other computer in the universe is also linear-bounded: the Turing Machine is just a handwavey abstraction for "if you have enough memory and time".
[2] A meta-optimizer is an optimizer of optimizers. Mesa- is the opposite of meta-, so it refers to the case in which an optimizer (read: gradient descent on a neural network) accidentally creates another optimizer with a different optimization strategy. In other words, it's optimizers all the way down.
This leads to a whole new set of alignment problems, called "inner-alignment problems", which means "the AI that is smarter than us and we can't trust created another AI that's smarter than it and it also can't trust".