Pausing AI Developments Isn't Enough. We Need to Shut It All Down

I’ve always found the dismissals of EY’s arguments to be pretty weak and people rarely engage on the actual arguments, even as his concerns have become more and more relevant with capability (in 2007 people thought AGI was at best 50yrs out and commonly people thought it was impossible, this was before any real deep learning success and GOFAI was a useless dead-end).

Most of the comments here are similarly dumb dismissals that don’t engage, and some even come across as highschool level mocking. It’s worth reading his A-Z book to at least understand why he holds his position.

Over time, the dumb responses from others make me think he’s probably more likely right than not. Given the extreme downside risks, I can understand why he argues this.

Humans don’t hate ants, they just have other goals.

In the case of an unaligned superintelligent AGI those goals may be something that just happens to satisfy its reward function but is otherwise “dumb” or unintentional (like making a lot of paper clips).

Intellectual capability does not get alignment for free.

What you see in the communicated text interface and the goals/system behind it are not the same (that cartoon with the smiley face), and we don’t understand how to evaluate the underlying state.

I don't think that's quite what he's saying, or at least the intended interpretation is a bit more nuanced.

It's more like... suppose you assign tiers of severity to various international policies; A-tier, B-tier, C-tier, etc. Different tiers require different enforcement mechanisms. For example, the international response to a country violating the "United Nations Declaration on Human Cloning" would be different than the international response to a country massively irradiating the atmosphere.

I think he's trying to imply that the proposed policy changes should be A-tier. It isn't a call for individual action or preemptive violence. He's describing a pre-requisite property that any adopted policy must fulfill to have a chance at being successful (according to his analysis of the situation).

On a related note, he is not saying that these policies are likely to happen or are even feasible. He's criticizing the policy proposals in the "Pause Giant AI Experiments" open letter, and describing what he thinks a real policy would have to look like.

It's increasingly easy and cheap to train GPTs and similar models. Pretty soon anyone who can pay for GPU compute will be able to do it, if that's not the case already. Even if every country in the world agreed to ban it, it still couldn't be stopped.

Do you mean the specifics of GPT-4, or Transformers in general ?