DeepMind says reinforcement learning is ‘enough’ to reach general AI

Anthropic principle

We still have unknown unknowns but we also know a lot more about how neural nets deal with various tasks and dataset preparations. We know what kind of applications are good enough and where they still fail, which is much more than a decade ago.

If you look at sci-fi movies with robots, they usually speak in a metallic voice but have good situational and language understanding. In reality it was the other way around, it's much easier to do artificial voices than understand the topic. That kind of naive understanding seems silly now, and this is how we gradually advance.

GPT-3 taught us that good sounding text is not that hard to generate if you have ample training data, but modeling the larger context is still hard. These kind of fine distinctions are what I call progress.

I completely agree that there is no conclusive evidence. However, it is the number one activity that influences cognitive performance in all animals that have been tested. Saying that sleep is not proven to be important for intelligence because the causal link has not been established seems a bit like saying exercise has nothing to do with muscle growth, because the full causal chain has also not been established (we do know a pretty full causal chain in this case, so maybe it is not the best example). I think if you were a betting man, and as a scientist you have to be to some degree, you would put your money on sleep supporting some essential process for intelligence.

That may be true, but that's not to simulate. "Dreaming" is used to "simulate", and it's only a hack, in humans probably not even explicitly programmed.

The hack is to have the simulator itself ... also be a learned system, and not to simulate the world, because you don't act on the real world, only a tiny part of it that you can measure and actually get into your brain (which is a simplified version of what you see, or a "latent variable"). There's no need to simulate anything that doesn't affect your reasoning. The information flows in reality with an intelligent actor is like this.

World (say, a tree falls) -> input representation (e.g. eyes) -> simplified version ("latent" version) -> intelligent actor -> muscles -> affects world.

Now what everybody thinks of as a simulator is something that simulates the whole thing. But if you insert one more link (output of reasoning agent at time T -> simplified representation at time T+1) you can then run a "simulation":

random simplified version ("what if your car became a tree and fell ?") -> intelligent actor -> next input for latent representation ("then what happens ?")

For safety reasons, it is probably prudent to disconnect the muscles in this state. You know, so you don't knock out your mother when you dream about boxing.

And as you say, this "predict the future" network is probably useful by itself in dealing with the world. So you can catch tennis balls and the like.

https://arxiv.org/abs/1803.10122

Or, if you like: https://www.youtube.com/watch?v=dPsXxLyqpfs

> But any sandbox we design won’t be good enough.

This isn't really true.

To bring back your analogy of searching the space of all programs - suppose we wanted to simulate every turing machine of at most N states for some sufficiently large N. One of these turing machines is going to encode an AGI. Nonetheless, it is still just an encoded turing machine being simulated by a turing machine simulator. No matter what great intelligence is encoded by those states, it's still a state machine following the same rules as every other turing machine in existence. It's a fixed set of states following a fixed set of rules on an unbounded-length tape of memory.

The same is true for an AGI in a computer program in a modern computer. A complex modern neural nets are still just a program that takes a giant array of numbers and feeds it through a network of transformations repeatedly, yielding a new array of numbers. No matter what brilliant information that array of numbers encodes, the array of numbers is still just a bunch of bits somewhere in RAM. It's data, not execution. It can't invoke the IPC stack unless *we* program functionality in that interprets the data and executes functionality based on it.

It's like the brain in a vat[0] thought experiment. A super-intelligent brain in a vat can't escape the lab unless the lab researchers give the brain control over something physical. A super-intelligent brain can't even know that it's a brain in the vat unless it is told. Why then would you expect a digital brain in a simulated environment to even know that it is what it is, much less how to perfectly execute a bug in a simulator (which might provably not exist[1]) to send a copy of its source code to another computer and end up replicating and distributing itself?

The simple answer is: Sandboxing an AI is significantly easier than you seem to think. When an AI takes over the world, it's going to be because someone handed the AI root access to a big tech data center so it can optimize operations; not because the AI discovered the perfect sequence of zero-day bugs to overcome all of humanity's best cybersecurity efforts.

[0] https://en.wikipedia.org/wiki/Brain_in_a_vat

[1] https://en.wikipedia.org/wiki/Formal_verification