DeepMind: A Generalist Agent

It always surprises me at the ease at which people jump on a) imminent AGI and b) human extinction in the face of AGI. Would love for someone to correct me / add information here to the contrary. Generalist here just refers to a "multi-faceted agent" vs "General" like AGI.

For a) - I see 2 main blockers,

1) A way to build second/third order reasoning systems that rely on intuitions that haven't already been fed into the training sets. The sheer amount of inputs a human baby sees and processes and knows how to apply at the right time is an unsolved problem. We don't have any ways to do this.

2) Deterministic reasoning towards outcomes. Most statistical models rely on "predicting" outputs, but I've seen very little work where the "end state" is coded into a model. Eg: a chatbot knowing that the right answer is "ordering a part from amazon" and guiding users towards it, and knowing how well its progressing to generate relevant outputs.

For (b) -- I doubt human extinction happens in any way that we can predict or guard against.

In my mind, it happens when autonomous systems optimizing reward functions to "stay alive" (by ordering fuel, making payments, investments etc) fail because of problems described above in (a) -- the inability to have deterministic rules baked into them to avoid global fail states in order to achieve local success states. (Eg, autonomous power plant increases output to solve for energy needs -> autonomous dam messes up something structural -> cascade effect into large swathes of arable land and homes destroyed).

Edit: These rules can't possibly all be encoded by humans - they have to be learned through evaluation of the world. And we have not only no way to parse this data at a global scale, but also develop systems that can stick to a guardrail.

Life has no purpose so clearly there is no rational reason to continue living/existing. A super-rational agent must know this.

I think that intelligence and emotions, in particular fear of death or desire to continue living, must evolve in parallel.

Serious question - why?

  I'm in the camp that thinks we're headed in a perpendicular direction and won't ever achieve powered flight with current efforts based on the simple idea that the basic tooling is wrong from first principles. I mean, most of the "progress" in flight has been due to getting better and learning how to understand a single piece of technology: fixed wing aircraft.

  A lot of recent powered flight findings have shown that real birds _don't_ just use fixed wings; in fact, they flap their wings! Until we start thinking from the ground up how to build and engineer systems that reflect the bird wing, we're essentially wandering around in the dark with perhaps only a piece of what we _think_ is needed for powered flight. (I'm not saying the bird wing is the best engineered thing for powered flight either, but I'm saying it's one of the best examples we have to model powered flight after and that notion has largely been ignored)

  I generally think it's hubris to spit in the face of 4 billion years of evolution thinking that some crafty fixed wing aircraft with X number more wingspan and horsepower will emerge magically as truly capable of powered flight - it will be a strange abomination at best.

A) Machine learning hasn't been focused on simple neural nets for quite some time.

B) There's no reason to believe that the organizational patterns that produce one general intelligence are the only ones capable of doing that. In fact it's almost certainly not the case.

By slowly iterating and using the best work and discarding the rest, we're essentially hyper-evolving our technology in the same way that natural selection does. It seems inevitable that we'll arrive at least at a convergent evolution of general intelligence, in a tiny fraction of the time it took on the first go-around!