Why are you so sure I'll lose?

However, I disagree with Yudkowsky in that we only get one shot at playing the game.

Here in the mucky real world, we're going to have a lot of chances at playing this game. The interaction between us and the AI is going to keep on going for thousands (?) of years and in millions (?) of different games.

Maybe it's because I've been reading a lot of philosophy recently. But I'm thinking that Yudkowsky is a bit too myopic here. Life and all it's sandy grit isn't even game-ifiable to me. There are no win/lose states. Yes, even death is variably considered a loss at many times in history. The entire idea of an 'end' here in the real world is not 'certain', or if that is even something to avoid.

I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing.

My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI.

Sorry for the long post hope this helped explain my point of view on the issue.

Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.