What Google DeepMind Means for A.I. • Hacker News Board

Re: What Google DeepMind Means for A.I.

#51

Post by ScottBurson » Wed, Mar 04, 2015, 11:51 PM UTC

karpathy wrote: post #17

I really like this line of work and I expect will grow quite substantially over the next few years. Of course, Reinforcement Learning has been around for a long time. Similarly, Q Learning (the core model in this paper) has been around a very long time. What is new is that normally you see these models applied to toy MDP problems with simple dynamics, and linear Q function approximations for fear of non-convergence e…

> Of these, the biggest and most insurmountable problem is the first one: Random exploration of actions. As humans we have complex intuitions and an internal model of the dynamics of the world. This allows us to plan out actions that are very likely to yield a reward, without flailing our arms around greedily, hoping to get rewards at random at some point.

In fairness, we weren't born with that model; we have to laboriously acquire it over a period of several years. An infant's flailings can look pretty random :-)

Re: What Google DeepMind Means for A.I.

#52

Post by rndn » Thu, Mar 05, 2015, 12:14 AM UTC

karpathy wrote: post #17

I really like this line of work and I expect will grow quite substantially over the next few years. Of course, Reinforcement Learning has been around for a long time. Similarly, Q Learning (the core model in this paper) has been around a very long time. What is new is that normally you see these models applied to toy MDP problems with simple dynamics, and linear Q function approximations for fear of non-convergence e…

What are the indications that a model of the world does not emerge somewhere in the neural networks of a Q Learning system?

Re: What Google DeepMind Means for A.I.

#53

Post by GenerocUsername » Thu, Mar 05, 2015, 12:38 AM UTC

_abattoir wrote: post #50

chockablock wrote: post #37

Earlier quoted context omitted.

>We experience the world at one second per second There is evidence from animal studies that the hippocampus (a brain structure critical for memory) can 'replay' remembered events at 10-20x speedup. See, for example: http://www.ncbi.nlm.nih.gov/m/pubmed/19709631/ Video at: http://youtu.be/Bv7zN2Or6Mg (Full-disclosure: I am the first author.) And in fact the OP uses biologically-inspired off-line replay as part of the…

Fascinating. Does this relate to the speed of dreams? i.e. a "dream" might seem to have taken hours when in fact the REM sequence was on the order of seconds?

Almost certainly.

In real time, the brain is processing all incoming stimulus.

In a dream/imagined scenario, you are only processing as much as you your brain is pushing into the scenario.

Your not collecting millions of photons via your eye and interpreting a ball as green, your brain just says "green ball" and moves on, allowing for much faster replays/dreams than real-world experience.

Re: What Google DeepMind Means for A.I.

#54

Post by raldi » Thu, Mar 05, 2015, 12:48 AM UTC

raldi wrote: post #28

Freeway isn't a driving game. It's a chicken-crossing-the-road game. I'm looking forward to the upcoming entry in next week's New Yorker "Corrections" section.

Um, holy shit, the article's been updated to refer to Freeway as "a chicken-crossing-the-road game", and there's a footnote acknowledging the correction.

Re: What Google DeepMind Means for A.I.

#55

Post by dicroce » Thu, Mar 05, 2015, 12:49 AM UTC

nsxwolf wrote: post #45

"the A.I. has not only become better than any human player but has also discovered a way to win that its creator never imagined." That's a pretty standard Breakout/Arkanoid technique - getting the ball behind the board and letting it do the work for you. Not knocking the AI, just nitpicking this writer.

That confused me too until I realized the author of the article was talking about the creator of the AI, not the game.

Re: What Google DeepMind Means for A.I.

#56

Post by astazangasta » Thu, Mar 05, 2015, 12:57 AM UTC

Mobiu5 wrote: post #48

This is actually pretty scary. It is basically giving AI a human-like form of will. It "desires" what you program it to desire and goes about achieving it, learning from its' own mistakes and becoming increasingly proficient at manipulating its' environment to achieve its' goal(s) along the way. It makes me excited, but also quite frightened to think what goals people might give AIs like this in the future...

This is a really poor rendering of "desire". If the word "desire" is meaningful, it has to be self-actualized. Otherwise it's just a programming condition, no different than a fuse box or a dead man's switch.

Re: What Google DeepMind Means for A.I.

#57

Post by darkmighty » Thu, Mar 05, 2015, 1:21 AM UTC

rndn wrote: post #52

karpathy wrote: post #17

I really like this line of work and I expect will grow quite substantially over the next few years. Of course, Reinforcement Learning has been around for a long time. Similarly, Q Learning (the core model in this paper) has been around a very long time. What is new is that normally you see these models applied to toy MDP problems with simple dynamics, and linear Q function approximations for fear of non-convergence e…

What are the indications that a model of the world does not emerge somewhere in the neural networks of a Q Learning system?

The system as shown has a state of a few frames. Suppose it had 1 instead for illustration. Suppose there's a dangerous ball right next to you, but it's actually moving away from you, so you don't need to run; in this model, you will probably learn to always run since you cannot know the velocity of the ball (whether it's coming at you). So this model could only work if you have a state-full representation of your world, which seems to be the case with simple games and a few frames, but is not the case in the real world (in the real world your naive state would need to arbitrarily back in time -- e.g. say you see a ticking time bomb, and then the clock is hidden from you).

This flaw is understandable since it seems Q-Learning was conceived to deal with Finite Markov Processes, which are finitely state-full, by definition (if you don't know them, they're essentially a non-deterministic state machine).

Re: What Google DeepMind Means for A.I.

#58

Post by Houshalter » Thu, Mar 05, 2015, 1:29 AM UTC

astazangasta wrote: post #56

Mobiu5 wrote: post #48

This is actually pretty scary. It is basically giving AI a human-like form of will. It "desires" what you program it to desire and goes about achieving it, learning from its' own mistakes and becoming increasingly proficient at manipulating its' environment to achieve its' goal(s) along the way. It makes me excited, but also quite frightened to think what goals people might give AIs like this in the future...

This is a really poor rendering of "desire". If the word "desire" is meaningful, it has to be self-actualized. Otherwise it's just a programming condition, no different than a fuse box or a dead man's switch.

It doesn't matter what word you use, the AI is still manipulating it's environment to achieve that "programming condition".

In any case, everyone's desires are programmed into them, just via genetics and evolution rather than humans and programming.

Re: What Google DeepMind Means for A.I.

#59

Post by darkmighty » Thu, Mar 05, 2015, 1:31 AM UTC

karpathy wrote: post #17

I really like this line of work and I expect will grow quite substantially over the next few years. Of course, Reinforcement Learning has been around for a long time. Similarly, Q Learning (the core model in this paper) has been around a very long time. What is new is that normally you see these models applied to toy MDP problems with simple dynamics, and linear Q function approximations for fear of non-convergence e…

Great answer!

Is there some exciting work being done to address those issues? (could you point to some?)

Re: What Google DeepMind Means for A.I.

#60

Post by NamTaf » Thu, Mar 05, 2015, 2:02 AM UTC

For all of the New Yorker's clout in journalism, sentences like the following make me wonder where their editors are. The run-on and sea of commas is atrocious! It's not the first time I've noticed this in the last few days either.

"Hassabis, who began working as a game designer in 1994, at the age of seventeen, and whose first project was the Golden Joystick-winning Theme Park, in which players got ahead by, among other things, hiring restroom-maintenance crews and oversalting snacks in order to boost beverage sales, is well aware that DeepMind’s current system, despite being state of the art, is at least five years away from being a decade behind the gaming curve."