Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

arxiv.org

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#31

Post by hyperbovine »

willwhitney wrote: post #16
gwern wrote: post #3

Note: it doesn't learn from pixels but features directly from RAM; and superhuman reaction time, with performance badly degrading when human-like delays added. Good discussions on Reddit: https://www.reddit.com/r/MachineLearning/comments/5vh4ae/r_a... https://www.reddit.com/r/smashbros/comments/5vin8x/beating_t...

Handling delays (and the uncertainty they entail) is a huge challenge, and I think it'll be a rich area of research. The simplest part of the problem is that delays in action or perception also slow the propagation of reward signals, and credit assignment is still a really hard problem. Thinking further afield, future models could learn to adapt their expectations to fit the behavior of a particular opponent. This ki…

Really naive question, can't they just train the net to react instantaneously on a $d$-delayed screen? I don't see conceptually why this approach would succeed with d=0 but fail for (say) d=25ms. (I am too busy/lazy to read the papers and understand what breaks down.)

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#32

Post by vladfi1 »

scythe wrote: post #14
revelation wrote: post #4

Earlier quoted context omitted.

we instead use features read from the game’s memory on each frame, consisting of each player’s position, velocity, and action state, along with several other values So it's cheating, presumably knowing the opponents action before the animation even starts to play.

Smash is played on analog displays precisely so that the lag between RAM and the display can be as small as possible, usually 50 ms. In fact there's a 50 ms delay added to the AI for this reason. However, the AI takes no account of the fact that it takes about 230 ms for a signal to travel from a human's retina through the occipital lobe and motor cortex and activate the motor neurons in the hand. The AI can also gen…

The bots are given a very reduced action set. The Falcon master bot actually had only cardinal directions on the control stick - later bots were also given the 4 diagonals.

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#33

Post by vladfi1 »

hyperbovine wrote: post #31
willwhitney wrote: post #16

Earlier quoted context omitted.

Handling delays (and the uncertainty they entail) is a huge challenge, and I think it'll be a rich area of research. The simplest part of the problem is that delays in action or perception also slow the propagation of reward signals, and credit assignment is still a really hard problem. Thinking further afield, future models could learn to adapt their expectations to fit the behavior of a particular opponent. This ki…

Really naive question, can't they just train the net to react instantaneously on a $d$-delayed screen? I don't see conceptually why this approach would succeed with d=0 but fail for (say) d=25ms. (I am too busy/lazy to read the papers and understand what breaks down.)

Basically we tried that and it sort of works, but performance degrades pretty fast with each frame of delay. The issue is likely that it makes credit assignment much harder. Instead of seeing an immediate change in the state (which your critic can interpret as good or bad), you have to wait a bunch of frames during which your previous actions are taking effect and interfering with the reward signal.

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#35

Post by erichocean »

stagbeetle wrote: post #25
revelation wrote: post #15

Earlier quoted context omitted.

You're wildly swinging between advantageous and not. No, this is just playing games. The ground rules must be clear: you get the screenshots and keyboard input in every frame, as a normal player. If the resulting AI sucks, who cares? Failure is part of doing science.

> You're wildly swinging between advantageous and not. What? > The ground rules must be clear: you get the screenshots and keyboard input in every frame, as a normal player. Perhaps if you want to start from flawed assumptions/ want to create an AI that's tweak-able to appear as human. Which would be pretty useful and practical for other applications, but not competitive play. We could go on and on about digital vs.…

> You give it all the tools it needs to beat a human player.

Okay, why not allow an NPC to just mess with the human player's actions then (blocking or delaying button clicks, for instance)? Surely, that falls into "all the tools", no?

IMO, the way you went about things isn't particularly compelling—your human opponents don't have white-box access to game internals, and if they did, guess what? They'd play better too.

So I agree with the GP: this is just playing games.

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#36

Post by vladfi1 »

SerLava wrote: post #6
gwern wrote: post #3

Note: it doesn't learn from pixels but features directly from RAM; and superhuman reaction time, with performance badly degrading when human-like delays added. Good discussions on Reddit: https://www.reddit.com/r/MachineLearning/comments/5vh4ae/r_a... https://www.reddit.com/r/smashbros/comments/5vin8x/beating_t...

This reminds me of Starcraft AI experiments. They can't actually make the computer smart, so they just jam 2000 button presses per second down the tube, giving every single unit its own simultaneous AI, and it out micromanages anyone. With Marines usually.

In starcraft there's a much bigger advantage since humans are inherently "single threaded", and so you can get much bigger discrepancies in APM (or EPM). Smash is more like 1 unit vs 1 unit micro. The precision and timing are still advantages for the AI, but not so much raw parallel compute.

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#37

Post by stagbeetle »

erichocean wrote: post #35
stagbeetle wrote: post #25

Earlier quoted context omitted.

> You're wildly swinging between advantageous and not. What? > The ground rules must be clear: you get the screenshots and keyboard input in every frame, as a normal player. Perhaps if you want to start from flawed assumptions/ want to create an AI that's tweak-able to appear as human. Which would be pretty useful and practical for other applications, but not competitive play. We could go on and on about digital vs.…

> You give it all the tools it needs to beat a human player. Okay, why not allow an NPC to just mess with the human player's actions then (blocking or delaying button clicks, for instance)? Surely, that falls into "all the tools", no? IMO, the way you went about things isn't particularly compelling—your human opponents don't have white-box access to game internals, and if they did, guess what? They'd play better too.…

> Okay, why not allow an NPC to just mess with the human player's actions then (blocking or delaying button clicks, for instance)? Surely, that falls into "all the tools", no?

Are you talking about physically delaying their inputs? As in from the controller to the main board? This would fall under the same category as a player hitting the controller out of his opponent's hand -- foul play.

> IMO, the way you went about things isn't particularly compelling—your human opponents don't have white-box access to game internals, and if they did, guess what? They'd play better too.

I'm not sure what exactly you're referring to here, but I'll respond to how I think you're trying to take this.

Source code wise: Yes. If the players had access to the source code the learning curve would be significantly shortened. Though, in due time, most would have figured out the mechanics fully, or within a short deviation, in closed source. A part of competitive play is this exact aspect. Players experimenting, sharing, and building up their understanding of the game. If the source was freely available to explore, most players would stick to the "show" part of the process, i.e working reflexes and learning combat -- what most elite players focus on (since they've mastered the science of the game already).

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#38

Post by avbor »

malikNF wrote: post #22
stcredzero wrote: post #19

Earlier quoted context omitted.

I could see this technology used for the bootstrapping of highly emergent MMO game worlds. It could be used to populate a world with fake "player" NPCs that are actually part of a simulated online ecosystem. Give the NPCs a large enough population, such that players cannot exert significant selection pressure, but give the NPCs real selection pressure through interaction with artificial life evolved with Genetic Algo…

League of legends for example has bots appear in PvP games. While these bots are not produced by the game's developers not a lot was done then to get rid of these things. I guess they were tolerated since it just make the queue times smaller for human players. ( http://boards.na.leagueoflegends.com/en/c/gameplay-balance/b... )

Well, Riot solved that by getting rid of Dominion. ;) But seriously, they were mainly present in the game mode mentioned, and now in Co-Op vs AI, just because they want to level the account and sell it. Doesn't really affect the majority of the population until selling point, and once it's been sold, it's easy to tell a botted account.

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#39

Post by avbor »

revelation wrote: post #15
stagbeetle wrote: post #7

Earlier quoted context omitted.

Part of the skill in competitive play is to be able to predict what move your opponent is going to do next. Most mid-level players already have a good grasp of prediction, which is arguably along the sames lines of being able to know with certainty what action your opponent is taking a few frames before he does it. Coupling that with pretty obscene frame-lag for Smash, it's not really that much of an advantage. As we…

You're wildly swinging between advantageous and not. No, this is just playing games. The ground rules must be clear: you get the screenshots and keyboard input in every frame, as a normal player. If the resulting AI sucks, who cares? Failure is part of doing science.

I disagree with this. I don't think it makes sense to make the problem space harder and include translating screenshots into a resulting keyboard input when you can look at the memory and figure out the same information. That's an entire problem of its own, and I really don't understand why people keep ignoring that. Or is the issue that with the amount of information being supplied, it's too revealing? In which case, would limiting the amount of data being passed in to just, map, positions of the players, their current moves, and direction of the current move be okay in your book? Or is even that too much information to give.

Re: Beating the World’s Best at Super Smash Bros. with Deep Reinforcement Learning

#40

Post by empath75 »

SerLava wrote: post #6
gwern wrote: post #3

Note: it doesn't learn from pixels but features directly from RAM; and superhuman reaction time, with performance badly degrading when human-like delays added. Good discussions on Reddit: https://www.reddit.com/r/MachineLearning/comments/5vh4ae/r_a... https://www.reddit.com/r/smashbros/comments/5vin8x/beating_t...

This reminds me of Starcraft AI experiments. They can't actually make the computer smart, so they just jam 2000 button presses per second down the tube, giving every single unit its own simultaneous AI, and it out micromanages anyone. With Marines usually.

That's an interesting definition of smart that doesn't including being able to manage hundreds of units simultaneously.
Post reply on HN