Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

arxiv.org

Post reply on HN
Print view View original

31–40 of 54 posts

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#31

Post by WhompingWindows »

s_m_t wrote: post #9
calebh wrote: post #6

For those who don't know Starcraft: microing refers to the small scale/precise maneuvering of units during a battle, as well as activating special unit abilities at certain times. In the lower level leagues microing is not as important as having a good overall large-scale "macro" strategy. The microing really starts coming into play at the higher levels where it can give the edge needed to win a battle. I am not very…

How can this be so? In my experience playing Starcraft the decision space for macro is very, very, very limited. You are essentially locked into a handful of viable build orders for the early game, a handful of timing attack opportunities in the early and mid game, and after that things veer so off track that very few players do more than wing it.

The problem here is the conflation of "macro" with "strategy". If the parent comment had defined the terms properly, there'd be less disagreement here. They rightly define micromanagement as the control and use of small numbers of units, whereas macromanagement refers to the management of the non-fighting, production related activities like producing workers constantly until saturated, producing supply depots on time, producing units constantly from the correct number of facilities, and so forth. Strategy itself refers to a broad game plan which may incorporate unit composition, large-scale plans for army movements, plans to limit the economy of the opponent, and so on.

Thus, strategy is the widest tent, and someone like sOs shows the truly massive number of strategies that can be employed. In contrast, macromanagement does indeed have a smaller decision space, since really the "decision" with macro is pre-defined. Make your economy and production as efficient as possible. There may be slight variations in how its done, especially in Brood War where the economics are much more nuanced than SC2. Nevertheless, if you watch a variety of pros macro out mass marine-marauder-medivac, they will usually have very very similar expansion timings, identical numbers of barracks, and any changes in the timings/build orders is strategic, not macro-related. There may be "on the fly" macro adjustments due to harassment damage, however strategy is far far more fluid of a category than macro.

When it comes to micro and decision space, however, I believe that it is mostly a matter of execution and not decision. We've seen perfect blink micro stalkers, dragoon AI bots, and I think it'd be far easier to take down top players in a 1v1 micro fight since computers are not limited by mouse accuracy, emotional concerns, stress, fatigue, and a number of other factors which affect your hands' abilities. In contrast, a top player should have a much better edge over a computer in strategic realms, since strategies must be tailored to the expected opponent's strategy and mind games heavily factor into this equation.

God, what a beautiful game.

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#32

Post by AmericanChopper »

taylorswift_ wrote: post #2

Interesting, outside of the academic merits of the research, I wonder if they could partner with Blizzard (or other similar RTS game studios) and start shipping games with AI that is very difficult to beat? That would usher in a new era of solo and offline game play that could be very fun and challenging! Plus we'd be training Skynet, err, I mean the neural network in the process!

A 'perfectly' micro-ing AI is both boring to implement and boring to play against. Somebody made one for SC2 back when that was popular, take a look at these videos to see how uninteresting it would become after the first attempt.

https://www.youtube.com/watch?v=0EYH-csTttw

https://www.youtube.com/watch?v=IKVFZ28ybQs

https://www.youtube.com/watch?v=DXUOWXidcY0

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#33

Post by WhompingWindows »

Creating a micromanagement AI is only the first step. This must be piped into a macro AI, which produces the units with with micromanagement can be done. Then, another critical layer beyond micro or macro is scouting the opponent and predicting what their plan is. Next, you need to adapt your own strategy, which must also be planned at the start of the match since you can't scout instantaneously or perfectly. Finally, your own initial strategy choice must be tailored to the map, to the player you're playing against, to recent games/strategies other pros have exposed as powerful/weak. There are an incredible number of layers of complexity to what a truly fierce Starcraft AI would need to do.

And yet, something similar may have been said about Go, Chess, Checkers, Poker, Jeopardy, or many other games before this. If enough talented minds and powerful devices focus on Starcraft, how would humans compete? Does anyone here think top pros would win in a best of 7 or a best of 11, for instance? Which top pros, and BW or SC2? To me, I just think it'd be arrogant to assume SC is different from all the other realms in which humans have been bested by machines.

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#34

Post by make3 »

AmericanChopper wrote: post #32
taylorswift_ wrote: post #2

Interesting, outside of the academic merits of the research, I wonder if they could partner with Blizzard (or other similar RTS game studios) and start shipping games with AI that is very difficult to beat? That would usher in a new era of solo and offline game play that could be very fun and challenging! Plus we'd be training Skynet, err, I mean the neural network in the process!

A 'perfectly' micro-ing AI is both boring to implement and boring to play against. Somebody made one for SC2 back when that was popular, take a look at these videos to see how uninteresting it would become after the first attempt. https://www.youtube.com/watch?v=0EYH-csTttw https://www.youtube.com/watch?v=IKVFZ28ybQs https://www.youtube.com/watch?v=DXUOWXidcY0

yeah but I'm pretty sure these bots still get crushed by humans because of macro, planning and positioning, which is why it takes top firms on the planet so long to solve the problem

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#35

Post by AmericanChopper »

make3 wrote: post #34
AmericanChopper wrote: post #32

Earlier quoted context omitted.

A 'perfectly' micro-ing AI is both boring to implement and boring to play against. Somebody made one for SC2 back when that was popular, take a look at these videos to see how uninteresting it would become after the first attempt. https://www.youtube.com/watch?v=0EYH-csTttw https://www.youtube.com/watch?v=IKVFZ28ybQs https://www.youtube.com/watch?v=DXUOWXidcY0

yeah but I'm pretty sure these bots still get crushed by humans because of macro, planning and positioning, which is why it takes top firms on the planet so long to solve the problem

If the AI wasn't very smart, then you'd expect there to be some optimal path that would beat them. But Starcraft has always been a game with very little skill ceiling on micro. Well micro'd units have the potential overcome huge macro advantages. A famous real-world example would be Boxer's Immortal Marines:

https://www.youtube.com/watch?v=WJp0t9n8DWk

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#36

Post by TulliusCicero »

WhompingWindows wrote: post #33

Creating a micromanagement AI is only the first step. This must be piped into a macro AI, which produces the units with with micromanagement can be done. Then, another critical layer beyond micro or macro is scouting the opponent and predicting what their plan is. Next, you need to adapt your own strategy, which must also be planned at the start of the match since you can't scout instantaneously or perfectly. Finally…

I don't think anyone doubts computers will eventually beat pros in Starcraft. I do think it will take quite a bit longer though, even with recent advancements seen in the likes of AlphaGo.

From a human perspective, Starcraft doesn't seem "harder" than Chess or Go, but it has more fluidity and ambiguity, both things computers struggle with. And the ruleset, as it were, is VASTLY more complicated.

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#37

Post by TulliusCicero »

s_m_t wrote: post #9
calebh wrote: post #6

For those who don't know Starcraft: microing refers to the small scale/precise maneuvering of units during a battle, as well as activating special unit abilities at certain times. In the lower level leagues microing is not as important as having a good overall large-scale "macro" strategy. The microing really starts coming into play at the higher levels where it can give the edge needed to win a battle. I am not very…

How can this be so? In my experience playing Starcraft the decision space for macro is very, very, very limited. You are essentially locked into a handful of viable build orders for the early game, a handful of timing attack opportunities in the early and mid game, and after that things veer so off track that very few players do more than wing it.

In practice there's a lot of small variations on each of those, depending on what your opponent is doing, the map, previous encounters, etc.

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#38

Post by TulliusCicero »

daemonk wrote: post #24
axxl wrote: post #5

Earlier quoted context omitted.

There is some stuff like this, you would see it on the ladder in SC2. Hacks that would perfectly split your marines against incoming banelings (they explode and do splash damage so the more you split the less effective they were), or auto burrowing/unburrowing roaches (which hides them from the enemy and they heal underground). It sounds cool but it’s way cooler when you saw a pro doing all that stuff themselves almo…

I appreciate the manual dexterity, but imagine the amount of strategy that AI assisted play would open up.

I agree with d1zzy, if you don't like the micro part of Starcraft, there are plenty of strategy games that cater to a more strategy-focused style of play.

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#39

Post by TulliusCicero »

AmericanChopper wrote: post #32
taylorswift_ wrote: post #2

Interesting, outside of the academic merits of the research, I wonder if they could partner with Blizzard (or other similar RTS game studios) and start shipping games with AI that is very difficult to beat? That would usher in a new era of solo and offline game play that could be very fun and challenging! Plus we'd be training Skynet, err, I mean the neural network in the process!

A 'perfectly' micro-ing AI is both boring to implement and boring to play against. Somebody made one for SC2 back when that was popular, take a look at these videos to see how uninteresting it would become after the first attempt. https://www.youtube.com/watch?v=0EYH-csTttw https://www.youtube.com/watch?v=IKVFZ28ybQs https://www.youtube.com/watch?v=DXUOWXidcY0

But even the best AIs in Starcraft are still quite bad compared to competitive players.

Re: StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer

#40

Post by jon_richards »

WhompingWindows wrote: post #33

Creating a micromanagement AI is only the first step. This must be piped into a macro AI, which produces the units with with micromanagement can be done. Then, another critical layer beyond micro or macro is scouting the opponent and predicting what their plan is. Next, you need to adapt your own strategy, which must also be planned at the start of the match since you can't scout instantaneously or perfectly. Finally…

SC2 has a lot of things that help the player over an AI, like imperfect information, a very large ruleset, and continuous movement and time.

The main advantage AI can exploit is "infinite micro" units. There are certain units in SC2 that are primarily limited by someone's ability to pay attention to them. Terran Reapers, for instance, in early patches could dance back and forth just within range Zerg Roaches, but out of range of being counterattacked (though the reapers would quickly die if player stopped paying attention). This required such fast movement that it was almost never employed except by professionals during the early game when not much was going on.

If an AI is not limited by the number of actions a person can take, these "infinite micro" units become extremely powerful. In particular, most of these types of units are within the Terran race and most powerful in the early game. The best AI in the near future will likely focus on early all-in attacks using these sorts of units (primarily reapers, widow mines, hellions, and medivacs). The human player would have to rely on defensive structures that would put them at a major disadvantage and then catch up in the mid-late game, relying on the fact that the AI gets worse as the game goes on and becomes less predictable.

Another thing to note is that Terran strategy relies quite heavily on killing the enemy workers in the early game, but the inbuilt controls for targeting workers are quite clunky and prioritise military targets. Players can target individual workers manually, but generally do so with their whole army at once, while an AI could select the perfect number of units required to 1-shot a worker, give them the command, then select another group, and so on.

Post reply on HN