Live data from Hacker News

DeepMind says reinforcement learning is ‘enough’ to reach general AI

venturebeat.com

21–30 of 312 posts

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#21
post #11

I'm more in Yann LeCun's camp who called unsupervised learning the cake, supervised learning the icing and reinforcement learning the cherry on top of the cake.

My opinion is that view is very simplistic and unnecessarily offensive to a whole class of researchers. MuZero, developed by David Silver, uses a combination of RL, supervised learning, and unsupervised learning (state representation) coupled with a planning algorithm. It accomplished things far beyond anything unsupervised learning can ever accomplish.

Unsupervised learning is exactly the wrong way to approach chess or other games that MuZero solves. It's also worth noting that traditional alpha-beta pruning + heuristics are basically neck and neck with the very best of neural network based techniques. I'll trust stockfish over a alpha-zero or MuZero for awhile longer if I'm trying to win a computer chess competition ...

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#23

Earlier quoted context omitted.

My opinion is that view is very simplistic and unnecessarily offensive to a whole class of researchers. MuZero, developed by David Silver, uses a combination of RL, supervised learning, and unsupervised learning (state representation) coupled with a planning algorithm. It accomplished things far beyond anything unsupervised learning can ever accomplish.

Unsupervised learning is exactly the wrong way to approach chess or other games that MuZero solves. It's also worth noting that traditional alpha-beta pruning + heuristics are basically neck and neck with the very best of neural network based techniques. I'll trust stockfish over a alpha-zero or MuZero for awhile longer if I'm trying to win a computer chess competition ...

Sure, Stockfish just uses millions of years of evolution to build its heuristics and can't be transferred to any other game. The point remains, calling RL a cherry on the cake compared to unsupervised learning when they are completely orthogonal and not mutually exclusive techniques is simplistic and unnecessarily offensive.

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#24

Earlier quoted context omitted.

My opinion is that view is very simplistic and unnecessarily offensive to a whole class of researchers. MuZero, developed by David Silver, uses a combination of RL, supervised learning, and unsupervised learning (state representation) coupled with a planning algorithm. It accomplished things far beyond anything unsupervised learning can ever accomplish.

Unsupervised learning is exactly the wrong way to approach chess or other games that MuZero solves. It's also worth noting that traditional alpha-beta pruning + heuristics are basically neck and neck with the very best of neural network based techniques. I'll trust stockfish over a alpha-zero or MuZero for awhile longer if I'm trying to win a computer chess competition ...

Not disputing your main point, but Stockfish now includes a neural network.

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#25
Are the just reformulating the principles of evolution in digital terms, and essentially not providing any new insights at all?

Yes, intelligence has been created by evolution. That doesn't imply that any system that is subject to evolutionary forces will lead to the creation of intelligence (and not within a reasonable timeframe, either). The challenge is to create a system that is capable of evolving intelligence.

Afaik some biologists even think that the evolution of intelligence was rather unlikely and would not necessarily happen again under the same circumstances as on earth.

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#26
post #3

Saying RL is sufficient to (eventually) achieve AGI is a bit misleading. One might similarly state that biological evolution is sufficient to (eventually) achieve biological general intelligence. Both statements are probably true, but the parenthetical (eventually) is doing an awful lot of heavy lifting.

RL + piggybacking on human culture might be enough, or evolution + RL for biological agents.

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#27

If you send a message in a bottle it will eventually land ashore somewhere, maybe in a century, who knows, and who knows whether it will be relevant by then or not, or civilization may not even exist by then, but sure it's similarly plausible to get to AGI via RL.

> send a message in a bottle

Just one - yes. But how about if you send millions of bottle messages?

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#28

This article is interesting, I even skimmed through their paper. But I think still the question remains: How to find the unified reward function? Or in other words, how to find answer to life? [It cannot be 42].

Intelligence would be produced in any Turing complete automata. But the universe has a frame rate of 10^34 (based on Planck constant.) We don't really have the tech to just run "evolution" of a universe or of even a psuedo biological substrate.

This seems far from clear. Just because a system is capable of turing complete computation does not imply that a generic state of the system will typically eventually produce intelligence or even something which is sophisticated in some sense.

As a trivial example, consider a variation of Conway's game of life which, in addition to black and white cells, also has green cells, where any cell next to one or more green cells will be a green cell in the next time step. A generic state in such a variation will have at least one green cell, and therefore all parts of it will eventually be green, and so no useful long running computation will be done, certainly none which takes where the green cells are into account. But, such a system would still be turing complete, because one could start in a state in which there are no green cells, and in those states you just have Conway's game of life.

That trivial example works as an existence proof, but even for less extreme cases it isn't clear. Consider ordinary conway's game of life. To paraphrase a question from Alex Flint on Alignment Forum (https://www.alignmentforum.org/posts/3SG4WbNPoP8fsuZgs/agenc... ) Suppose we have some 10^50 by 10^50 square where an agent is supposed to be implemented, and this 10^50 by 10^50 square is at the top left corner of a, say, 10^100 by 10^100 square, where the rest of the square is initialized randomly, is it even possible for the agent to be such that it has a high chance of successfully influencing the large scale state of the rest of the 10^100 by 10^100 region in the way that is desired? It isn't clear. It isn't clear that a structure can withstand the interactions with a surrounding chaotic region. Perhaps some systems are such that they do allow Turing-complete computation, and are such that typical states result in complex behavior, but are also such that all really structured behavior is always very "fragile", and can only continue in a structured way if what interacts with it is in a small set of possible interactions.

To be capable of Turing complete computation, is not, I think, sufficient for "life" (a self-maintaining thing) to arise from typical/generic states, even when under the assumption that typical/generic states lead to continually complex behavior (to exclude the spreading green cells case)

Also, I don't think we can confidently say that the Plank time is "the universal frame rate". Better to refer to Bremermann's limit and the Margolus–Levitin theorem , though these bounds depend on the amount of energy available. (10^33 operations per second per joule, where the energy is the average energy of the system doing the computation)

Re: DeepMind says reinforcement learning is ‘enough’ to reach general AI

#29
post #9

good luck with that. DeepMind should sponsor B. F. Skinner award, to honor the father of their behaviorist theories of 'reward and punishment' as a sort of all-encompassing theory of everything related to cognition. At least now they are torturing GPUs and not some poor lab animals. on a serious note the only positive outcome of all this shameless PR is that the heavy investment in ML/RL might trickle down to actual…

> towards understanding natural intelligence, a prerequisite for creating an artificial one.

I've thought about this before, and I'm not convinced it's really prerequisite. Naturally developed intelligence in my mind may actually be highly constrained and inefficient because it was limited to what was biologically feasible. i.e. There may be simpler ways of achieving comparable results. Natural intelligence does however have the benefit of being an actual working model, but deciphering the blackbox may be just as hard as developing a working theory from first principles.

Post reply on HN