Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaGo Zero: Learning from scratch

deepmind.com

Post reply on HN
Print view View original

191–200 of 324 posts

Re: AlphaGo Zero: Learning from scratch

#191

Post by nojvek »

namelost wrote: post #143
romaniv wrote: post #113

Earlier quoted context omitted.

> They have a new reinforcement learning algorithm that should be generically applicable to anything where a long sequence of moves results in a specifically gradable outcome. Statements like these always make me wonder why certain obvious things weren't tried. If it's so generic, why wasn't it tried on Chess? Or was it tried, failed to impress and thus didn't make it into the press release? This is a big problem wit…

The conventional wisdom for Chess engines is that aggressive pruning doesn't work well. Chess is much more tactical than Go, selective algorithms tend to lead to some crucial tactic being missed, and the greater the search depth, the more likely that is. Modern Chess engines are designed to brute-force the search tree as efficiently as possible. I will go out on a limb here and say they would wipe the floor with Alph…

Until I see AlphaGo zero defeating StockFish 100-0 and with same algorithm defeating best Go AI and killing the Atari games including montezuma’s revenge, I call this hype bullshit.

Give me your results on OpenAI gym in a variety of different styles of games including GTA and WoW. I will believe you if a generic unsupervised algorithm running on a single machine is absolutely destroying the best players.

Until then ...

Re: AlphaGo Zero: Learning from scratch

#192

Post by nothis »

ericb wrote: post #7

I'm reminded of Eliezer Yudkowski's article "There is no fire Alarm for Artificial General Intelligence." Is this smoke? https://intelligence.org/2017/10/13/fire-alarm/ Yes, this is not an AGI. But the hockey-stick takeoff from defeats some players, to defeats an undefeated world-champion, to defeats the version of itself that beat the world champion 100% of the time is nuts . If this happens in other domains, like f…

I'm sure it's naive to jump to sci-fi conclusions just yet, but I admit it's equal parts fascinating and terrifying. The general message of the posts is that human knowledge is cute but not required to find new insights. Define the measure of success and momma AI will find the answer. At this point, the path to AGI is about who first defines its goals right and that seems... doable? Even scarier: We think the holy grail of AI is simulating a human being. The AI of the future might chuckle at that notion.

Re: AlphaGo Zero: Learning from scratch

#193

Post by sobellian »

The catch is that this isn't quite zero human knowledge, since the tree search algorithm is a human discovery, and not one that came easily to humans. It also massively cuts down on the search space for an appropriate policy function.

That means that this setup isn't necessarily general. How applicable is MCTS to games with asymmetric information, a la Starcraft? What about games that can't quite be modeled with an alternating turn-based game tree like bughouse?

Re: AlphaGo Zero: Learning from scratch

#194

Post by goatlover »

nojvek wrote: post #191
namelost wrote: post #143

Earlier quoted context omitted.

The conventional wisdom for Chess engines is that aggressive pruning doesn't work well. Chess is much more tactical than Go, selective algorithms tend to lead to some crucial tactic being missed, and the greater the search depth, the more likely that is. Modern Chess engines are designed to brute-force the search tree as efficiently as possible. I will go out on a limb here and say they would wipe the floor with Alph…

Until I see AlphaGo zero defeating StockFish 100-0 and with same algorithm defeating best Go AI and killing the Atari games including montezuma’s revenge, I call this hype bullshit. Give me your results on OpenAI gym in a variety of different styles of games including GTA and WoW. I will believe you if a generic unsupervised algorithm running on a single machine is absolutely destroying the best players. Until then .…

Add Pacman and Pitfall to the list. Humans have played perfect games of both. My understanding is DeepMind performed poorly on those games.

Re: AlphaGo Zero: Learning from scratch

#195

Post by nothis »

goatlover wrote: post #176
tim333 wrote: post #166

Slightly scary how it went from zero to superhuman play in three days. I wonder if general AI will go that way one day.

General AI relative to an individual human, or billions of humans? The sum total of human beings, or organizations of humans is superhuman relative to an individual. We've had superhuman organizations for millennia. I'm not sure how much general AI will be different, other than the large scale automation of jobs which would happen. As Rodney Brooks pointed out, all technology happens within a context, not a vacuum. A…

One of the more interesting things the success of "starting with zero" suggests is that the idea that some mystical "human consciousness" is the end goal for AI might be laughable in the long term. AI might just casually bypass human consciousness, say "oh, hi!" and wave us goodbye a day later. Also, a factor of 7 billion "happens" in computer science.

This is getting rather creepy to think of, even if it's still science fiction. At this point, I could see a computer that out-thinks humanity within decades. What would it think? What would we even do with its findings? Would we understand it? Would it understand itself? Would it know how to manipulate us?

Re: AlphaGo Zero: Learning from scratch

#196

Post by anomnomidas »

sobellian wrote: post #193

The catch is that this isn't quite zero human knowledge, since the tree search algorithm is a human discovery, and not one that came easily to humans. It also massively cuts down on the search space for an appropriate policy function. That means that this setup isn't necessarily general. How applicable is MCTS to games with asymmetric information, a la Starcraft? What about games that can't quite be modeled with an a…

There's a Dota 2 bot by OpenAI that played games with itself and managed to beat a lot of pros in the scene. It's still SF mid only no runes and some restricted items, but it shows that there is also potential for Starcraft.

https://blog.openai.com/dota-2/

Re: AlphaGo Zero: Learning from scratch

#197

Post by tyler_larson »

sobellian wrote: post #193

The catch is that this isn't quite zero human knowledge, since the tree search algorithm is a human discovery, and not one that came easily to humans. It also massively cuts down on the search space for an appropriate policy function. That means that this setup isn't necessarily general. How applicable is MCTS to games with asymmetric information, a la Starcraft? What about games that can't quite be modeled with an a…

That's not quite what they're talking about WRT zero human knowledge.

The problem is that there's no intrinsic scoring system for Go, nothing specific to maximize, so it's difficult to tell a computer whether a given outcome is "good" or "bad". So early versions of AlphaGo used a collection of human-played Go games to get an idea of what constitutes "good" and what is "bad", so it can then train its model to predict whether a move will make things better or worse.

This new system forgoes that step, and instead has the model play itself starting at random and looking for patterns that end up winning games. It's as if you gave the rules to the game of Go to a culture that's never heard of it before, and they evolved their own play style entirely in isolation.

Their result is a model that is better than the one that was developed with human influence, and that's the interesting bit.

Re: AlphaGo Zero: Learning from scratch

#198

Post by tc »

When things will start getting interesting is when we figure out how to get move simulation and search into the network itself, rather than programming that on the outside. As far as I know, no-one has even the faintest idea of how to do that. We have an existence proof that this should be possible.

The networks are great at perception and snap-prediction. Anything a human can do in 200ms is fair game. And with clever engineering, we can make magic happen by iterating or integrating those things.

But it's after that first 200ms that humans get really intelligent. When we can come up with an architecture that lets the networks themselves start simulating possibilities, backtracking, deciding when to answer now or to think more -- when the network owns the loop -- then it will get interesting.

Re: AlphaGo Zero: Learning from scratch

#199

Post by shmageggy »

xoroshiro wrote: post #178
espadrine wrote: post #134

Earlier quoted context omitted.

Random rollouts are what the MC in MCTS stands for. Without that, it is simply a tree search. Excerpt from the paper: > [AlphaGo Zero] uses a simpler tree search that relies upon this single neural network to evaluate positions and sample moves, without performing any Monte-Carlo rollouts.

Does this mean it learns what to search? I wonder why they thought it was a good idea. I thought the whole point of MC was that pruning algorithms like the ones in chess wouldn't work for a larger search space.

That's what the policy is. Given a board state, the policy gives you a distribution over all available moves.

Re: AlphaGo Zero: Learning from scratch

#200

Post by gwern »

mannigfaltig wrote: post #163
jtolmar wrote: post #54

Earlier quoted context omitted.

Yes, but tree search + neural net is still pretty generic. It only assumes that you can enumerate branches.

It also presumes that one can simulate the world at low cost. In AlphaGo Zero it takes 0.4 s for 1.600 node extensions, but in this case the cost of the world is negligible. Anyway, assuming you need that many node extensions to get decent quality updates, that puts a rather a tight limit on the cost of simulating the world.

DM has already done a bunch of work on 'deep models' of environments to plan over. Use them and you have 'model-predictive control' and planning, and this tree extension to policy gradients would work as well (probably). It could be pretty interesting to see what would happen if you tried that sort of hybrid on ALE.
Post reply on HN