Why traditional reinforcement learning will probably not yield AGI [pdf] • Hacker News Board

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#31

Post by gnulinux » Wed, Oct 21, 2020, 9:28 PM UTC

LatteLazy wrote: post #22

People love talking about intelligence but I'm yet to see a measurable definition. The best anyone can manage is "you know it when you see it" (like porn). Am I missing something or is this all just a dark road to an unknown destination (possibly just a dead end?)

The value of definability is moot at best imho. "Mathematics" is also not easy to define, there are volumes philosophers of mathematics discussing this, some even say "mathematics is what mathematicians find interesting" (similar to your porn example) but this doesn't stop us from studying mathematics. Same goes for science e.g.. People like Popper or Kuhn spent a lot of mental cycles arguing what is and what is not science, yet people still do science every day without reading them.

In some ways this way of thinking is too meta. In order to be a good mathematician, it is not necessary to understand the nature of mathematics from an outside perspective. This view can be very useful e.g. if you're working on foundations, but that doesn't mean before being good at mathematics one must be good at understanding the nature of mathematics. That seems like the job of philosophers, not mathematicians. (Well, sometimes the set has intersections, e.g. Brouwer, Hilbert, Godel and Penrose (theo. physicist) wrote some works on phil. of math).

EDIT: To express this slightly more formally: in order to understand a theory of a model, you do not need to understand the model comprehensively. You can be an expert in intelligence science by studying the falsifiable and predictive theories of intelligence science without understanding the nature of intelligence itself.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#32

Post by benchaney » Wed, Oct 21, 2020, 9:30 PM UTC

gnulinux wrote: post #26

bra-ket wrote: post #19

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient mode…

> The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop. I'm a layman (just a software engineer) but am curious, I train my cat only with rewards (never punishment because apparently doesn't work on cats) and the kitty learned how to high-five me, sit, jump, follow me etc. It seems to work really well for us. Basically, ever time he does something desirable, I cli…

Your cat was already capable cognition before you started training it. GP is talking about generating a cognition where it did not previously exist.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#33

Post by bra-ket » Wed, Oct 21, 2020, 9:30 PM UTC

anotherProd wrote: post #23

bra-ket wrote: post #19

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient mode…

As someone who's a software engineer (not data scientist) but is interested in consciousness and by extension AGI and dabbled in some ML algorithms, I find it surprising how often I see the sentiments of AGI being possible or impossible using some sort of algorithm. Obviously I could be missing some great breadth and depth of research (there's definitely a lot I don't know) but from what I've read "we have no idea" i…

My point was that RL/DL is being used like some kind of massive hammer to hit all the nails. Cognition requires different, specialized, energy-efficient tools.

> consciousness

All talk about this is premature and "pre-science", before we figure out more basic, fundamental things like object storage and recall from memory, object recognition from sensory input, concept representation and formation, the exact mechanism of "chunking" [1], "translational invariance" [2], generalization along concept hierarchy and different scales, representation of causal structures, proximity search and heuristics, innate coordinate system, innate "grammar".

Even having a working, biologically-plausible model of navigation in 3d spaces by mice, without spending a ton of energy training the model, would be a good first step. In fact there is evidence that navigational capacity [3] is the basis of more abstract forms of thinking.

On all of these things we have decades worth of research and widely published, fundamental, Nobel-winning discoveries which are almost completely ignored by the AI field stuck in its comfort zone. Saying "we have no idea" is just being lazy.

Edit: As for OP's actual paper I think something like complex-valued RL [4] might bypass his main claims entirely. But my point is that RL itself is a dead end, trivializing the problem at hand.

[1] https://en.wikipedia.org/wiki/Chunking_(psychology)

[2] http://www.moreisdifferent.com/2017/09/hinton-whats-wrong-wi...

[3] http://www.scholarpedia.org/article/Grid_cells

[4] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22c...

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#34

Post by [deleted] » Wed, Oct 21, 2020, 9:34 PM UTC

benchaney wrote: post #32

gnulinux wrote: post #26

Earlier quoted context omitted.

> The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop. I'm a layman (just a software engineer) but am curious, I train my cat only with rewards (never punishment because apparently doesn't work on cats) and the kitty learned how to high-five me, sit, jump, follow me etc. It seems to work really well for us. Basically, ever time he does something desirable, I cli…

Your cat was already capable cognition before you started training it. GP is talking about generating a cognition where it did not previously exist.

[deleted]

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#35

Post by ForHackernews » Wed, Oct 21, 2020, 9:40 PM UTC

GregarianChild wrote: post #27

bra-ket wrote: post #19

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient mode…

Reinforcement learning is Turing complete [1], so if AI is possible at all, then it can be realised through RL. cognitive psychology You are overselling the insights of this discipline. Has cognitive psychology solved its replication problems? Where is the world-beating AI that is based on "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience? [1] https://arxiv.org/…

> Reinforcement learning is Turing complete [1], so if AI is possible at all, then it can be realised through RL.

This seems like overstating your point. Nobody has been able to rigorously define "AI" yet, so there's no way of saying whether it's possible with a Turing machine architecture. The human brain, at least, doesn't seem that similar to a Turing architecture. Neurons don't carry out anything like discrete operations.

Maybe it's possible to run AGI on a Turing machine, maybe it's not, but there are more options than simply "possible with a Turing machine" or "completely impossible".

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#36

Post by qyv » Wed, Oct 21, 2020, 9:44 PM UTC

postnihilism wrote: post #25

bra-ket wrote: post #19

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient mode…

> "This 'Skinnerism' has been discredited in cognitive psychology decades ago and makes absolutely no biological sense whatsoever for the simple reason that any organism trying to adapt in this way will be eaten by predators before minimizing its "error function" sufficiently." > "Living learning organisms have limited resources (energy and time), and they cut the search space drastically through shortcuts and heuris…

> But those heuristics and hardcoded biases were developed through brute force optimization over the course of billions of years, a massive amount of energy input and many organisms being devoured.

This is true in the context of the universe as a whole, not by the organism itself.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#37

Post by sebastos » Wed, Oct 21, 2020, 9:46 PM UTC

lostmsu wrote: post #15

xamuel wrote: post #5

Earlier quoted context omitted.

Yes you can, but said representations will necessarily be misleading. It's illuminating to consider Big-O notations: why don't we "simplify", since real numbers are so much easier, why don't we declare, e.g., that O(n) is "1", O(n^2) is "2", etc.? Well, then what should O(2^n) be? A million, perhaps? But then what about O(n^1000000)? To be consistent, you'd have to say O(n^1000000) was something below a million, sinc…

I don't see how representation of floats by natural numbers that we all use on daily basis is misleading. In fact, your entire comment is just an ordered sequence of natural numbers, and it does not seem to be very misleading (though it tried to trick). Besides, the article is purposed to be a rigorous mathematical proof of current representations in RL being unsuitable for AGI, but I haven't seen a Definition for "m…

?? Current reinforcement learning algorithms don't score an action by producing a line of text that is read by a human. It assigns a value by writing a floating point number into a register on the computer somewhere. If it were otherwise, and there was some kind of interpretation unit that was needed to compare two rewards and decide which one was bigger, then hey, you're not using the reals for your rewards anymore, just like the paper suggested you'd have to.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#38

Post by IshKebab » Wed, Oct 21, 2020, 9:51 PM UTC

xamuel wrote: post #7

nx2059 wrote: post #3

Earlier quoted context omitted.

The article was pretty over my head, but does your argument hold to the various augmented neural net systems such as Neural Turing Machines, or Differentiable Neural Computers?

The argument isn't so much about the type of agent (which I think is what Neural Turing Machines etc. are about), it's about the type of environment. In traditional Reinforcement Learning, environments give real-number-valued rewards (or even rational-number-valued rewards which is even more constrained). Presumably this was a decision that was made with hardly a second thought because real numbers are most familiar…

But isn't the sophisticated structure what leads to the real number. If you change the rewards to something more complex surely you still have to pick between actions and at some point you'll have to evaluate which one is "better" and I can't see why you couldn't use real numbers to represent utility.

I mean, humans are general intelligences, and you can translate pretty much any human reward into money, which is a real number.

The paper is super long though. Maybe someone can give a TL;DR that makes sense.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#39

Post by joe_the_user » Wed, Oct 21, 2020, 9:51 PM UTC

benchaney wrote: post #32

gnulinux wrote: post #26

Earlier quoted context omitted.

> The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop. I'm a layman (just a software engineer) but am curious, I train my cat only with rewards (never punishment because apparently doesn't work on cats) and the kitty learned how to high-five me, sit, jump, follow me etc. It seems to work really well for us. Basically, ever time he does something desirable, I cli…

Your cat was already capable cognition before you started training it. GP is talking about generating a cognition where it did not previously exist.

Why couldn't a mechanism (say A Neural Turing or whatever) that you train be "cognition capable" when you start and then be trained to actual behavior after that?

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#40

Post by kypro » Wed, Oct 21, 2020, 9:52 PM UTC

gnulinux wrote: post #26

bra-ket wrote: post #19

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient mode…

> The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop. I'm a layman (just a software engineer) but am curious, I train my cat only with rewards (never punishment because apparently doesn't work on cats) and the kitty learned how to high-five me, sit, jump, follow me etc. It seems to work really well for us. Basically, ever time he does something desirable, I cli…

Also a layman, but I think OPs point wasn't that it isn't possible, but that's it's not effective or analogous to how humans or other species learn.

For example, your cat's brain isn't just a randomly initialised neural net. Your cat comes pre-wired in such a way that it understands certain things about its environment and has certain innate biases that allow you to train it to do simple tricks with relative ease through a reward mechanism.

A more analogous example would be building a cat-like robot with four legs and a neural processor then switching it on and expecting to be able to train it with treats. Without a useful initial neural state (founded with an understanding of cognitive psychology and neuroscience) it would be almost totally useless.