Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Reinforcement Learning to Play StarCraft

arxiv.org

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Deep Reinforcement Learning to Play StarCraft

#41

Post by daveguy »

raus22 wrote: post #29
aab0 wrote: post #23

Earlier quoted context omitted.

"The researchers found that attempting to move at a superhuman pace (eg one action every frame), resulted in a subpar performance." Moving at extremely fine-grained timesteps can make learning much more difficult, because now a reward arrives millions of timesteps delayed rather than hundreds or thousands. It's like trying to teach a NN to compose piano music by starting down at the 1ms raw audio level. This is part…

Why not go the other way and decrease the actions per minute so you learn the overall point of the game , And with each game the actions per minute increases.

That sounds like an interesting research angle. The thing about AI research is there are so many open ends there are essentially unlimited research options. If you can pose it as a problem and identify a reasonable programming approach then you have an avenue for AI research. Deep Learning isn't the end of AI research. It is the beginning.

Re: Deep Reinforcement Learning to Play StarCraft

#42

Post by comex »

raus22 wrote: post #29
aab0 wrote: post #23

Earlier quoted context omitted.

"The researchers found that attempting to move at a superhuman pace (eg one action every frame), resulted in a subpar performance." Moving at extremely fine-grained timesteps can make learning much more difficult, because now a reward arrives millions of timesteps delayed rather than hundreds or thousands. It's like trying to teach a NN to compose piano music by starting down at the 1ms raw audio level. This is part…

Why not go the other way and decrease the actions per minute so you learn the overall point of the game , And with each game the actions per minute increases.

Or maybe extend the traditional categories of macro and micro with another one, call it 'nano'... the micro agent indicates where each unit ought to be in 9 frames, and the nano agent figures out how to take them there. Since the timescale is so short, the agent could brute-force enumerate possible moves to some extent and figure out which is optimal, like chess AI. Or use a separate network.

I guess that's inelegant when a deep network already has its own concept of fine-grained versus coarse-grained layers, and should be able to do this on its own with the right training method.

Re: Deep Reinforcement Learning to Play StarCraft

#43

Post by ktRolster »

whorleater wrote: post #31
ufo wrote: post #30

Earlier quoted context omitted.

In addition to the training data, there is also the BWAPI project, that lets the bots play the game against other bots or humans. http://bwapi.github.io/ There isn't something similar available for SC2 due to a mix of technical and nontechnical issues: https://github.com/bwapi/bwapi/wiki/FAQ#will-there-be-an-api...

Yep! But as someone that built something on top of the BW API, I'd wager anyone going after SC1 AI to probably write their own thing. It's still an amazing API for general heuristics and modeling, but there's a few issues with it that stand in the way of making it a scaleable foundation.

What is wrong with it?

Re: Deep Reinforcement Learning to Play StarCraft

#44

Post by ktRolster »

h4nkoslo wrote: post #13

It's a misconception that StarCraft is a strategy game. If you look at how it's actually played by human pros, it looks closer to a fighting game; very reflex-driven & heavy on micro-interactions. You would expect an un-gated AI with effectively infinite actions per second to do very well.

After the most recent top Starcraft tournament (gsl code S), they asked the loser why he lost. He said, "I worked hard to prepare for this tournament, but my opponent prepared better."

What kind of things did he prepare? It wasn't reflexes, it was strategy. What kind of strategies was he preparing? He watched his opponent's past games, and came up with some build orders of course, but in this case, the primary strategy he came up with was an army composition hoping to counter what his opponent had been doing recently. When the opponent had the proper counter to that strategy, he won the rest of the games easily.

Re: Deep Reinforcement Learning to Play StarCraft

#45

Post by daodedickinson »

Merem wrote: post #35
rfrey wrote: post #34

Earlier quoted context omitted.

I am not a gamer... but I have to ask. 780 APM is possible with human players? 13 actions per second? I can't come close to clicking my mouse that fast. I don't think I can blink that fast.

Yes, it's possible. Bear in mind that not just clicks count though. July managed a peak APM of 818 in Stracraft: Brood War in an official game. The average APM of a match is usually between 300 and 500, also depending on the race that is played.

I think that may be high for an average... and to give an idea of how much APM is spam, there's that Stork vs. Idra replay from the World Cyber Games where Stork demolishes Idra despite using 70 APM to Idra's 300+. Of course, Protoss takes the least APM but Stork may have been the least spammy player.

Re: Deep Reinforcement Learning to Play StarCraft

#46

Post by pcr0 »

rfrey wrote: post #34
wamatt wrote: post #28

Earlier quoted context omitted.

That's a good point and not really correct to conflate the two. It would then seem to suggest, that forces with more than say, 5 units (780 average sustained APM) and above, would likely be getting into super human territory.

I am not a gamer... but I have to ask. 780 APM is possible with human players? 13 actions per second? I can't come close to clicking my mouse that fast. I don't think I can blink that fast.

I found this video demonstrating ~300 APM, which is quite insane already. I think >500 APM is probably burst speeds, not sustained.

https://www.youtube.com/watch?v=zmYhX8fjmo8

Re: Deep Reinforcement Learning to Play StarCraft

#47

Post by knicholes »

ajsalminen wrote: post #39
knicholes wrote: post #33

Earlier quoted context omitted.

You've got to be kidding. There's one going on right now with a prize pool of $31k! http://wiki.teamliquid.net/starcraft/Main_Page

Are you referring to the just completed Afreeca Starleague with $21k prize pool? Only the top two got more than a typical month's paycheck (winner did get over $10k) and it lasted a couple of months. There also haven't been any other events even close to that size this year for BW. Pro players don't typically live off tournament winnings.

That's the one! The winner pulled in $13,500 USD. I must have misinterpreted "paycheck" in your comment "nobody is getting a paycheck for playing competitive games."

[edit] Ah, $21k, not $31k. Thanks. [edit] And yes, it was recently completed and isn't actually ongoing. Not sure what I was smoking when I made so many false statements.

[edit] Maybe I was thinking the tournament starting October 28th, http://wiki.teamliquid.net/starcraft/VANT36.5_National_Starl... for $33k total prize pool.

Re: Deep Reinforcement Learning to Play StarCraft

#48

Post by nickpsecurity »

danielvf wrote: post #14
pkfrank wrote: post #9

Earlier quoted context omitted.

Wow, I hadn't seen this before. Here is "Automaton 2000" controlling 20 marines vs 40 banelings, without losing a single unit. https://youtu.be/DXUOWXidcY0?t=52 Pretty cool.

And as cool as that is, this is even more terrifying, as a hundred zergslings dodge seige tank cannons and destroy them. https://youtu.be/IKVFZ28ybQs It's enough to make you scared for the future of humanity.

What... the... hell..!? That's basically what they show on movies where the action stars have superhuman movement. Except, it's zergling's perfectly coordinating the demolition of siege tanks. Awesome demo of AI micro.

Re: Deep Reinforcement Learning to Play StarCraft

#49

Post by BirdieNZ »

empath75 wrote: post #40
Merem wrote: post #35

Earlier quoted context omitted.

Yes, it's possible. Bear in mind that not just clicks count though. July managed a peak APM of 818 in Stracraft: Brood War in an official game. The average APM of a match is usually between 300 and 500, also depending on the race that is played.

A lot of that is 'spamming', where they keep their hands moving and doing things even if they're not particularly meaningful, just so they can keep the pace of the eye/brain/hands feedback loop going.

Maybe early on it is spamming, but later on in the game most actions because useful actions. Having said that, some APM measurements only tracked effective APM (EAPM), which was clicks that actually achieved some result rather than merely selecting a unit.

Re: Deep Reinforcement Learning to Play StarCraft

#50

Post by posterboy »

h4nkoslo wrote: post #13

It's a misconception that StarCraft is a strategy game. If you look at how it's actually played by human pros, it looks closer to a fighting game; very reflex-driven & heavy on micro-interactions. You would expect an un-gated AI with effectively infinite actions per second to do very well.

It's a misconception that fighting games aren't strategy games. The term that conflates both is tactics, if I got my vernacular right. Ofc, strength and reactivity is a tactical advantage.
Post reply on HN