Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why traditional reinforcement learning will probably not yield AGI [pdf]

philpapers.org

Post reply on HN
Print view View original

111–120 of 123 posts

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#111

Post by YeGoblynQueenne »

mannykannot wrote: post #110
YeGoblynQueenne wrote: post #109

Earlier quoted context omitted.

Ah, so you are making a formal argument? In that case you should stick to formal language. And probably publish it in a different venue :)

No, I am simply responding to your rather formal point, in kind. Unless you are aguing for it being an established fact that the time evolution took to produce intelligent life rules out any form of reinforcement learning producing AI in any remotely reasonable period of time, then that original point of yours does not seem to be going anywhere. In your work on theorem proving, am I right in guessing that there are n…

I didn't make a formal point- my comment is a comment on an internet message board, where it's very unlikely to find formal arguments being made. But perhaps we do not agree on what constitutes a "(rather) formal point"? I made a point in informal language and in a casual manner and as part of an informal discussion ... on Hacker News. We are not going to prove or disprove any theorems here.

But, to be sure, as is common when this kind of informal conversation suddendly sprouts semi-formal language, like "argument", "claim", "proof", "necessarily follows" etc, I am not even sure what exactly it is we are arguing about, anymore. What exactly is your disagreement with my comment? Could you please explain?

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#112

Post by mannykannot »

YeGoblynQueenne wrote: post #111
mannykannot wrote: post #110

Earlier quoted context omitted.

No, I am simply responding to your rather formal point, in kind. Unless you are aguing for it being an established fact that the time evolution took to produce intelligent life rules out any form of reinforcement learning producing AI in any remotely reasonable period of time, then that original point of yours does not seem to be going anywhere. In your work on theorem proving, am I right in guessing that there are n…

I didn't make a formal point- my comment is a comment on an internet message board, where it's very unlikely to find formal arguments being made. But perhaps we do not agree on what constitutes a "(rather) formal point"? I made a point in informal language and in a casual manner and as part of an informal discussion ... on Hacker News. We are not going to prove or disprove any theorems here. But, to be sure, as is co…

"Necessarily" has general usage as well, you know... why would you read it otherwise, especially given the reasonable observation you make about this site? And my original point is not actually wrong, either: whether reinforcement learning will proceed at the pace of evolution is a topic of speculation - it is possible that it will, and possible that it will not.

Insofar is I have an issue with your comment, it is that it is not going anywhere, as I explained in my previous post.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#113

Post by xamuel »

gwern wrote: post #106
xamuel wrote: post #65

Earlier quoted context omitted.

Hi, thanks for looking at my paper. If you're interested in the relation between Lucas-Penrose stuff and enumeratability of ordinal codes, you might like I.J. Good (1969), "Godel's theorem is a red herring" (2 pages). Can you elaborate on what it is about my paper that strikes you as crankery? I'm a fan of yours so it would be much appreciated.

Sadly, I was only able to count through a vanishingly small subset of the reals in a finite time, and therefore am not a general intelligence, and so it would be foolish of me, a machine made of a handful of atoms, to try to criticize this paper. It sure would be nice if I could appreciate music, but you've proven that's impossible, so it is what it is.

Whether man is machine can't be trivially answered using a few handwavy applications of Godel's theorem nor of observations about the structure of the reals. My paper does not make any attempt to weigh on that question, neither does it claim nor imply that humans have any supernatural powers of enumerating reals or ordinals, and you grossly misrepresent me by implying it does.

Rather, my paper is on the less ambitious question of whether the traditional RL model (with its real-valued rewards) accurately captures the full set of reward-giving environments an AGI should be capable of comprehending.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#114

Post by xamuel »

visarga wrote: post #105
xamuel wrote: post #7

Earlier quoted context omitted.

The argument isn't so much about the type of agent (which I think is what Neural Turing Machines etc. are about), it's about the type of environment. In traditional Reinforcement Learning, environments give real-number-valued rewards (or even rational-number-valued rewards which is even more constrained). Presumably this was a decision that was made with hardly a second thought because real numbers are most familiar…

There are multi-reward agents and multi-task agents but all rewards get added into a final scalar value. And gradient based methods need to have this one scalar value to derive gradients from. Since you mentioned higher dimensional representations for rewards, I want to remind that the sub-fields of Inverse RL and Model-based RL are concerned with reward representation and prediction by neural nets. Also, it doesn't…

I make no attempt to disprove an entire field, rather to question an implicit assumption, namely that real number rewards are flexible enough to capture all relevant environments an AGI should be able to comprehend. Quite the opposite, I indicate ways reinforcement learning could be modified to get past the roadblock I point out with real numbers.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#115

Post by xamuel »

lostmsu wrote: post #104
sebastos wrote: post #37

Earlier quoted context omitted.

?? Current reinforcement learning algorithms don't score an action by producing a line of text that is read by a human. It assigns a value by writing a floating point number into a register on the computer somewhere. If it were otherwise, and there was some kind of interpretation unit that was needed to compare two rewards and decide which one was bigger, then hey, you're not using the reals for your rewards anymore,…

In that interpretation the article basically says "you can't precisely optimize function F in domain X with by optimizing function F' in domain Y, when cardinality of Y Well, obviously! That is basically the definition of cardinality!

The argument in the paper has nothing to do with cardinality.

Review my comment a few steps above about Big-O. The reasoning there shows the real numbers, despite having cardinality of the continuum, are inadequate for measuring even the following countable set of Big-O complexity classes: O(2^n) together with O(n),O(n^2),...,O(n^i),...

Is there something special about intelligence that implies we only need to consider RL-environments with real-valued rewards, and not, say, big-O-complexity-class rewards? Maybe there is. But if there is, the proof will have to be nontrivial---at a bare minimum, it should make some reference to what the reals are (e.g., the unique complete ordered field). This would be quite world-shattering, to real analysts if to no-one else. But none of the RL literature seems to make any reference to what the reals actually are, instead just taking it for granted that the reals are a magical number system as flexible as anybody could ever want. (Other authors essentially pointed this out before I did; see footnote 10 in my paper.)

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#116

Post by lostmsu »

xamuel wrote: post #115
lostmsu wrote: post #104

Earlier quoted context omitted.

In that interpretation the article basically says "you can't precisely optimize function F in domain X with by optimizing function F' in domain Y, when cardinality of Y Well, obviously! That is basically the definition of cardinality!

The argument in the paper has nothing to do with cardinality. Review my comment a few steps above about Big-O. The reasoning there shows the real numbers, despite having cardinality of the continuum, are inadequate for measuring even the following countable set of Big-O complexity classes: O(2^n) together with O(n),O(n^2),...,O(n^i),... Is there something special about intelligence that implies we only need to consid…

I re-read your Big-O comment, and it still does not make sense. The fact, that mapping all n^i to arctan(i) and i^n to arctan(i) + pi is somehow "misleading" is not making it theoretically impossible for RL algorithms from finding an optimal solution in that metric. If for a specific instance you're saying n^1e300 is wrong because it is impractical, and 2^n would be preferred, you simply posed the original task incorrectly by asking for a wrong metric.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#117

Post by xamuel »

lostmsu wrote: post #116
xamuel wrote: post #115

Earlier quoted context omitted.

The argument in the paper has nothing to do with cardinality. Review my comment a few steps above about Big-O. The reasoning there shows the real numbers, despite having cardinality of the continuum, are inadequate for measuring even the following countable set of Big-O complexity classes: O(2^n) together with O(n),O(n^2),...,O(n^i),... Is there something special about intelligence that implies we only need to consid…

I re-read your Big-O comment, and it still does not make sense. The fact, that mapping all n^i to arctan(i) and i^n to arctan(i) + pi is somehow "misleading" is not making it theoretically impossible for RL algorithms from finding an optimal solution in that metric. If for a specific instance you're saying n^1e300 is wrong because it is impractical, and 2^n would be preferred, you simply posed the original task incor…

It's not about whether the agent will or will not find an optimal solution (no agent can possibly find good solutions in every environment). It's about whether the agent could even understand the true environment based on your real-value-reward description of it.

Suppose the true environment has various tasks the agent can do which give big-O-complexity-valued rewards, one task giving a reward of O(2^n), and others giving rewards of O(n^i) for various i. For concreteness, say Task A rewards O(2^n), Task B rewards O(n^10000), and Task C rewards O(n^20000).

Now suppose you present this to the agent using real-valued rewards, say, where O(n^i) is replaced by arctan(i) and O(2^n) is replaced by arctan(2)+pi, as you suggest, then the agent will be deluded into thinking, e.g., that Task B and Task C give almost identical rewards (Task B gives reward 1.57069633 and Task C gives reward 1.57074633, which barely differ from each other at all). This is misleading because in the true environment, Task C gives much more reward than Task B. Yes, the agent understands Task C gives a bigger reward, but the agent totally mis-understands how much bigger :)

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#118

Post by dkersten »

mannykannot wrote: post #108
dkersten wrote: post #101

Earlier quoted context omitted.

Not grounds for stopping work[1], but perhaps grounds to explore other avenues[2] to see if something else might yield faster results. I’m no expert, but my personal opinion is that AGI will probably be some hybrid approach that uses some reinforcement learning mixed with other techniques. At the very least, I think an AGI will need to exist in an interactive environment rather than just trained on preset datasets. P…

Fair enough, though I do not think the evidence from evolution moves the needle much with respect to the timeline. For one thing, evolution was not dedicated to the achievement of intelligence.

Sounds reasonable.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#119

Post by YeGoblynQueenne »

mannykannot wrote: post #112
YeGoblynQueenne wrote: post #111

Earlier quoted context omitted.

I didn't make a formal point- my comment is a comment on an internet message board, where it's very unlikely to find formal arguments being made. But perhaps we do not agree on what constitutes a "(rather) formal point"? I made a point in informal language and in a casual manner and as part of an informal discussion ... on Hacker News. We are not going to prove or disprove any theorems here. But, to be sure, as is co…

"Necessarily" has general usage as well, you know... why would you read it otherwise, especially given the reasonable observation you make about this site? And my original point is not actually wrong, either: whether reinforcement learning will proceed at the pace of evolution is a topic of speculation - it is possible that it will, and possible that it will not. Insofar is I have an issue with your comment, it is th…

>> Insofar is I have an issue with your comment, it is that it is not going anywhere, as I explained in my previous post.

I see this god-moding of my comment as a pretend-polite way to tell me I'm takling nonsense, that seems to be designed to avoid criticism for being rude to one's interlocutor on a site that has strong norms against that sort of thing, but without really trying to understand why those norms exist, i.e. because they make for more productive conversations and less wasting of everyone's time.

You made a comment to say that unless I claim that X (which you came up with), then my comment is not going anywhere. The intellectually corteous and honest response to a comment with which one does not agree is to try and understand the reasoning of the comment. Not to claim that there is only one possible explanation and therefore the comment must be wrong. That is just a straw man in sheep's clothing.

And this is not surprising given that it comes at the heels of nitpicking about supposedly important terminology (necessarily!). This is how discussions like this one go, very often. And that's why they should be avoided, because they just waste everyone's time.

Re: Why traditional reinforcement learning will probably not yield AGI [pdf]

#120

Post by lostmsu »

xamuel wrote: post #117
lostmsu wrote: post #116

Earlier quoted context omitted.

I re-read your Big-O comment, and it still does not make sense. The fact, that mapping all n^i to arctan(i) and i^n to arctan(i) + pi is somehow "misleading" is not making it theoretically impossible for RL algorithms from finding an optimal solution in that metric. If for a specific instance you're saying n^1e300 is wrong because it is impractical, and 2^n would be preferred, you simply posed the original task incor…

It's not about whether the agent will or will not find an optimal solution (no agent can possibly find good solutions in every environment). It's about whether the agent could even understand the true environment based on your real-value-reward description of it. Suppose the true environment has various tasks the agent can do which give big-O-complexity-valued rewards, one task giving a reward of O(2^n), and others g…

Your paper does not prove anything in regards to what agent "mis-undestands". "Understanding" is not a mathematical concept in this case. It would still progressively find more and more optimal solutions to this task, so I see no problem there.
Post reply on HN