> That's quite a bad argument when it comes to "existential risk for all of humanity".
It's not a bad argument. It's the standard for creating one. Engineers get paid to resolve risk/problems. So, if you claim there is one and it is fantastically an 'existential risk for all of humanity', one should at least be able to explain how exactly it exists in the most basic technical terms. I haven't heard a single example of this and in my work it was resolved from focusing on the hard problem : What is intelligence?.
> Of course, so far no one has any idea how pragmatically it could play out.
Then there is no argument. Let it go. What you don't understand is that millions/billions are at play over this argument. Humans love to make up fantastical nonsense/games for profit. You're being played. If someone truly claims they care about this issue, why haven't they done the first step? Define how exactly this pragmatically could play out. Were talking about people with PhDs, industry titans, and millions if not billions are at play and you're telling me that between them all they can't come up with a signle pragmatic way this scenario plays out? It's a scam that's why.
> Maybe it'll start with something as banal as VisualStudio + Cortana providing an advantage to AI design, that helps designing a better Cortana, that helps designing a better Cortana, that ...
Garbage in garbage out. Weak AI in Weak AI out.
> Yes, of course, it's unlikely in the coming next 1-2 decade, because the best "AGI" we know (the human mind), is a hodgepodge of specialized faculties jury rigged together by ancient code lossy compressed and stored in a multi-stage bootloading wetware that takes years to learn reading the magnificent ~26 letters of a latin alphabet (but similarly takes the same time to learn walking, talking, hearing, rudimentary thinking, a minimal concept of self, a theory of mind and so on, and for some reason it has a shitty API, it must go through all of the aforementioned to even begin learning chess).
And if I told you the foundation of AGI has already been defined, the core computational model proofed, and that a functional system exists today how would anything change? AGI is here and there is no doomsday playing out. It's running on an 8-core processor, a GPU, and 32GB of RAM completing interactive exercises and going through a series of tests defined by its creator. The bowl of spaghetti above your head was decoded and a functional equivalent instantiated in computer hardware. The locks are compiled binaries, read only code regions, and a power button.
> So far AI is focusing on computer vision, some NLP, (strategy or easy to evaluate) games. At least the current progress/results are clustered around those areas. We don't know when will be the next big leap, maybe in knowledge representation, or goal formation, maybe in other kinds of very useful, more general intelligence related areas.
Instead imagine a child AGI with no bearings on the world... The human equivalent eons ago... Being carefully cultivated and learned in a controlled environment. No takeoff intelligence scenario. No end of the world. A child simply stumbling its way through controlled scenarios being oversee and tweaked by its creator. Not so scary huh?