Why traditional reinforcement learning will probably not yield AGI [pdf]
1–10 of 123 posts
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#2Author here. One particularly topical observation (topical because HN has recently featured discussions about ways to merge statistical and symbolic approaches to AI), from Section 4.2.3: certain cutting-edge number systems such as Conway's "surreal numbers" are so sophisticated that they require lots of symbolic logic just to do basic operations with. Of course this makes these number systems hard to work with, but they do not suffer the flaws pointed out in the paper, which the easier-to-work-with real numbers suffer. Since surreal numbers inherently require symbolic logical methods to work with, it follows that any sort of statistics-based RL agent for environments with surreal number rewards, would automatically combine symbolic logic and statistics.
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#3Author here. One particularly topical observation (topical because HN has recently featured discussions about ways to merge statistical and symbolic approaches to AI), from Section 4.2.3: certain cutting-edge number systems such as Conway's "surreal numbers" are so sophisticated that they require lots of symbolic logic just to do basic operations with. Of course this makes these number systems hard to work with, but…
The article was pretty over my head, but does your argument hold to the various augmented neural net systems such as Neural Turing Machines, or Differentiable Neural Computers?
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#4The argument is totally flawed and therefore BS. It rests on a clearly wrong assumption, that you can't represent number sets with higher cardinality (there are more real numbers, than natural numbers) using the sets of lower cardinality.
You can use natural numbers to represent real numbers with arbitrary precision. You can also use natural numbers to represent non-Archimedian number systems.
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#5The argument is totally flawed and therefore BS. It rests on a clearly wrong assumption, that you can't represent number sets with higher cardinality (there are more real numbers, than natural numbers) using the sets of lower cardinality. You can use natural numbers to represent real numbers with arbitrary precision. You can also use natural numbers to represent non-Archimedian number systems.
Yes you can, but said representations will necessarily be misleading. It's illuminating to consider Big-O notations: why don't we "simplify", since real numbers are so much easier, why don't we declare, e.g., that O(n) is "1", O(n^2) is "2", etc.? Well, then what should O(2^n) be? A million, perhaps? But then what about O(n^1000000)? To be consistent, you'd have to say O(n^1000000) was something below a million, since that complexity level is below O(2^n). And then whatever you chose for that, that would constrain you to pack O(n^billion), O(n^trillion), etc., into smaller and smaller ranges of possible numbers. You'd inevitably be forced to eventually assign almost identical numbers to, say, O(n^(2^100)) and O(n^(2^999)), which is misleading because those are not at all close to each other in complexity.
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#6The argument is totally flawed and therefore BS. It rests on a clearly wrong assumption, that you can't represent number sets with higher cardinality (there are more real numbers, than natural numbers) using the sets of lower cardinality. You can use natural numbers to represent real numbers with arbitrary precision. You can also use natural numbers to represent non-Archimedian number systems.
No, non-Archimedian means transfinite arithmetic (or nonstandard reals, depending on whether you're looking for discrete or continuous).
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#7Author here. One particularly topical observation (topical because HN has recently featured discussions about ways to merge statistical and symbolic approaches to AI), from Section 4.2.3: certain cutting-edge number systems such as Conway's "surreal numbers" are so sophisticated that they require lots of symbolic logic just to do basic operations with. Of course this makes these number systems hard to work with, but…
The article was pretty over my head, but does your argument hold to the various augmented neural net systems such as Neural Turing Machines, or Differentiable Neural Computers?
The argument isn't so much about the type of agent (which I think is what Neural Turing Machines etc. are about), it's about the type of environment. In traditional Reinforcement Learning, environments give real-number-valued rewards (or even rational-number-valued rewards which is even more constrained). Presumably this was a decision that was made with hardly a second thought because real numbers are most familiar to people... but such "appeals to familiarity" are totally irrelevant for such an alien field as AGI :) The point of the paper is that a genuine AGI should be able to comprehend environments that involve rewards with a more sophisticated structure than can be accurately represented using real numbers.
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#8Although I have only skimmed the paper, I think it's kinda trying to say (although someone with a better mathematical background than me might poke me for this) that the reward hypothesis (http://www.incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewa...) - the notion that every goal or purpose can be framed as the maximization of a real-valued function - isn't really applicable for most of the time. This is quite intuitively agreeable even without the math - do we really think that the many things we do in our lives were perfromed to optimize an "oracle" loss function? Our human mind is comprised of ridiculously complex systems of neurons and cells that generates a variety of emergent behaviors, and saying that those emergent behaviors are actually a solution of a very complex optimization problem is very, very bold. Often the reward functions are just abstractions of what we perceive (although they aren't entirely useless - keep in mind that all models are wrong but some are useful).
Although the paper is trying to say that the real number system isn't robust enough to express the goal/purpose of more complicated, "abstract" tasks, it speculates that a higher-order number system (such as the hyperreal or surreal numbers) would be able to achieve this. I currently disagree with this view - I view of "intelligence" as we know of today more as emergent phenomena of complex systems of autonomous agents (in the case of human intelligence, the emergent phenomena of neurons and other cells interacting with the external world), but that's a topic for another day.
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#9Earlier quoted context omitted.
The article was pretty over my head, but does your argument hold to the various augmented neural net systems such as Neural Turing Machines, or Differentiable Neural Computers?
The argument isn't so much about the type of agent (which I think is what Neural Turing Machines etc. are about), it's about the type of environment. In traditional Reinforcement Learning, environments give real-number-valued rewards (or even rational-number-valued rewards which is even more constrained). Presumably this was a decision that was made with hardly a second thought because real numbers are most familiar…
IIUC, the claim is that the very idea of a (real valued) “objective function” to be “optimized” is broken?
Re: Why traditional reinforcement learning will probably not yield AGI [pdf]
#10Although I have only skimmed the paper, I think it's kinda trying to say (although someone with a better mathematical background than me might poke me for this) that the reward hypothesis ( http://www.incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewa... ) - the notion that every goal or purpose can be framed as the maximization of a real-valued function - isn't really applicable for most of the time. This is quite in…
I think you understood the basic gist of the paper quite well, that's a good way of describing it, and thanks for the link.
>it speculates that a higher-order number system (such as the hyperreal or surreal numbers) would be able to achieve this
I didn't mean to give that impression, sorry if it came off that way. Rather, what I say is that those number systems don't suffer the particular flaw that the real numbers suffer. There might still be other flaws. That's why in the beginning of Section 4 I wrote: "There are at least two potential ways to change RL so as to make it applicable to such tasks and, thus, at least potentially capable of leading to AGI. Of course, there is no guarantee that removing the roadblock in this paper will cause RL to lead to AGI. There might be other roadblocks besides the inadequate reward number system"