Live data from Hacker News

AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

deepmind.com

321–330 of 459 posts

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#321

Earlier quoted context omitted.

One macro technique used by AlphaStar agents that is not used by human pros is building extra workers beyond currently exploitable capacity. This gives them reserves when attacked and some workers killed. They can also ramp up mining at a new base quickly by moving the extra workers there. Apparently the benefits outweigh the costs for these workers for AlphaStar. It will be interesting to see if some pros decide to…

Workers mine 40 minerals per minute and cost 50, taking... 15 seconds to build? I forget. Workers beyond 24 provide zero benefit (better to send them to the natural). Let's say you make 4 extra at a cost of 200 minerals and then lose 4 workers to harassment. You are out 200 minerals in both cases, but the prebuilt workers in the prebuilt case will mine an extra... 100 minerals? (40 + 30 + 20 + 10). This doesn't take…

The numbers you cite are close enough that your estimations are good to work with (12 seconds to build, closer to 60 minerals at full efficiency but down to 40 for probes #17-24, etc)

The extra workers aspect was the most interesting decision-based adjustment AlphaStar made on conventional pro level wisdom of "standard" play. It has a couple of factors in play, that I trust the AI factored in and more and tested over several games for its long-term benefit to winning a game:

- every 8 probes you build requires a pylon as well. total cost of 500 minerals

- workers are safer in the main than in an unoccupied natural (long distance mining) to harassment and pressure

- when your expansion completes, having 4 workers vs 8 workers vs 16 workers potentially has huge impact to the immediate spike in income

- what you mention -- the prebuilt workers will dampen the impact of most worker harassment to purely the resource cost of the lost workers.

My guess was that well executed harassment by an opponent in practice games put AlphaStar in very limited situations with a crippled economy that it couldn't fight its way out of, so this was a catch-all harassment "counter" -- it's ok if you kill a few probes, at least it won't throw off my economy completely and I can still continue my overall gameplan.

After that I think the next most important aspect was planning ahead for a bigger income spike when their expansion was done without waiting to build out another 16 workers after the nexus was ready.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#322
post #146

Earlier quoted context omitted.

The results are obviously impressive, but even then there is a lot of work to do as far as learning efficiency goes: "The AlphaStar league was run for 14 days, using 16 TPUs for each agent. During training, each agent experienced up to 200 years of real-time StarCraft play. " MaNa probably played less than 2-3 years of Starcraft in his whole life (by that I mean 24hr x 365d x 3), and was learning with a much less foc…

Another way to think about it is that a human brain is mostly doing transfer-learning, on top of a 99%-baked deep net that was wired up during foetal development from our DNA, where that DNA-persisted model has "seen" hundreds of millions of years of training data. Humans don't have to learn to process, recognize, and classify objects in visual sense-data, for example. We can do that from the moment we're born, becau…

Human genome isn't even a gigabyte of data. That's less than a byte per neuron and a big chunk of that data actually has to go into "how to make a kidney cell" and "which way to route veins". So while some basics have to be hard-coded, it can't be remotely close to "99% transfer from ancestors".

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#323
post #108

Earlier quoted context omitted.

That time, the ai didn’t really even try to engage. In fact, the ending of the match was marked by the entirely absent group of stalkers as the natural was engaged. It’s likely safer to say the AI was confused in general at that point, possibly related to the camera change, but we didn’t really get to see the quality of stalker micro that game

"possibly related to the camera change, but we didn’t really get to see the quality of stalker micro that game " In software, changes in assumptions can break what depended on them. There could be many assumptions in its neural net centered on full visibility. They should probably retrain all or just some from scratch with the camera change in from the beginning to see what happens. Then, it will be firmly encoded in…

They mentioned that they retrained after the camera change and it was equivalent to the AIs that beat Mana 5-0 by their metrics.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#324
post #41

Every year or so we get another huge advance... Well, more accurately, something comes along to benchmark the state of AI research against a human activity. Then come the HN comments. For Alpha Go: Oh this is impressive but can't generalize. Wake me up when it doesn't have to have information precoded/doesn't learn from human players For alpha go 0: So this is cool but not amazing because they're all perfect informat…

I was impressed with DeepMind's work on Go and Chess. It seemed to truly grasp the strategy and "understand" the games better than any human, which is in stark contrast to previous engines that relied on brute-force tactical brilliance. AlphaStar plays like I'd expect a computer to play. It makes some very stupid decisions that suggest a lack of real "understanding" or strategic thinking, such as its wacky unit compo…

We tend to think in hierarchical terms: tactics, strategy, and the like. It allows us to quickly create imperfect viable solutions. AlphaStar is probably lacking such distinction and sees the game as a sequence of actions. So it takes a lot more time for it to learn behavior patterns that we call strategy.

I think after training for maybe 20 thousand in-game years it will have decent strategy, while still lacking a real "understanding" of what a strategy is.

We probably will not see humanlike learning speed and adaptability before development of methods for learning hierarchical representations.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#325
post #46

This is really impressive, I didn't expect starcraft to be played this well by a machine learning based AI. I'm excited to read the paper when it comes out! That said, I'm not sure I agree that it was winning mainly due to better decision making. For context, I've been ranked in the top 0.1% of players and beaten pros in Starcraft 2, and also work as a machine learning engineer. The stalker micro in particular looked…

Also these wins are not using same inputs that human receive (ie on screen image) and outputs that humans are allowed. They instead use PySC APIs which has much more flexibility, perfect information and no constraints of limited screen real estate and pixels. There is a claim in that article that they have another version being trained that uses on screen only information but I still don’t know if AI is allowed to by…

Perfect information is a bit of a stretch. There was still fog of war. The AI just played as if the portion of the map visible and actionable at any point in time was the whole map. They retrained with a restriction to a given locus of attention that can change, akin to a screen the player is looking at and acting on.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#326
post #46

This is really impressive, I didn't expect starcraft to be played this well by a machine learning based AI. I'm excited to read the paper when it comes out! That said, I'm not sure I agree that it was winning mainly due to better decision making. For context, I've been ranked in the top 0.1% of players and beaten pros in Starcraft 2, and also work as a machine learning engineer. The stalker micro in particular looked…

Yeah that stalker micro really showcases a particular advantage leveraged by the AI.

I'd love to watch the results of constraining the AI so instead of seeing the whole map at once it has to pan around the same way a human would to get updated information on each battle. Counting those "info-gathering" window pans against the actions tally might yield slightly fairer APM metrics. (EDIT: Turns out they built a new agent for game 11 to do just that)

One of my biggest beefs with strategy games of this genre occurred around the time sprites went 3D and the player viewports got smaller (presumably to showcase all the cosmetic detail, and since it became harder to distinguish between visuals when zoomed out farther). I always feel too constrained on the modern games - like I can't see enough of the map at once. In my opinion that "full size viewport" gives a multi-tasking edge to the engine that the player doesn't share (beyond the human cognitive overhead from context switching you already pointed out).

On the other hand I find it fascinating our AI's have become strong enough at our games that we're having to handicap them to avoid players crying foul that they're not fair.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#327
APM is a really really misleading metric to use here. Most starcraft pros spam keys to keep themselves loose and ready when the time comes. Even at the start of a game, you'd often see players with 500 apm warming up the fingers.

Here, there is the laughable graph of the computer apm over time. The key points here is that when there were the mass battles that won the game, the apm spiked to >1000. And if you look closely in slow motion, there was perfect split targeting. A human player wouldn't be able to perfectly select the exact number of stalkers to hit the enemy without wasting a surplus shot. They can, but not when it's mass stalkers. This efficiency is just beyond humans. The APM here indicates much more effective use of an action than a typical human.

This is super impressive as an achievement, but this is clearly not a smarter ai, but moreso an ai like the video a while ago where zerglings could perfectly micro against siege tanks to avoid splash damage. It is clearly better than humans in certain ways, but not smarter.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#328
post #298
post #80

Earlier quoted context omitted.

Perfect micro bots don't excite me much, because they've existed all along, and it's not an AI task.

The interesting part to me is that, as far as I understand, the AI figured out this strategy by itself, basically deciding that it would be a good way for it to win games, rather than being specifically programmed to do it. That's actually pretty cool! Other than that, I agree, and am also much more interested in what happens when you have a more level playing field (using camera movement rather than API, limiting re…

It was not programmed to do the thing, but all these tactics were in seed replays, from which the agent started its learning. So, it actually not figured the move _by itself_, only found it useful.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#329

Earlier quoted context omitted.

You aren't kidding about the stalkers. Check out the bar chart at the bottom of the page: https://deepmind.com/blog/alphastar-mastering-real-time-stra... I guess it makes sense that the AI would favor such a micro-heavy unit. I imagine it would be a nightmare to deal with perfect blinking.

Dealing with perfect blinking is basically impossible, since you can blink back your units right before they die. Stalkers are balanced around the fact that HUMANS have limits to how well they can micro.

While the "skill cap" on blink stalkers is extremely high, there are many hard counters that can stop even perfect blink micro. MaNa won because he went for one these. Immortals are the perfect hard counter to stalkers because

- cost-for-cost, they are more efficient in a faceoff (resources)

- immortals are space-efficient dps (damage per second) in a battle. In a given battle, an army of 4 immortals is far more likely to all be in range of an enemy and doing damage than an army of 8 stalkers bumping against each other trying to get to the priority target

- immortal shots do not have projectiles, but are instant. No matter how perfect your stalker control, once an immortal targets a stalker, it is guaranteed to take 30+% of its hitpoints in damage.

The last point is very important. Once MaNa had 3+ immortals, even with perfect blink micro, a little bit of target fire and timing micro on MaNa's part allowed him to slaughter the stalker army one stalker per volley, while it takes them longer to clean up the immortals (especially with shield battery support).

Another thing glossed over in this discussion -- AlphaStar did more than classic blink micro. It did a very technical maneuver (the casters briefly allude to it) of triggering the barrier on one immortal with a single laser, then focusing all fire on an immortal whose barrier was already down from a previous iteration of this tactic, and then walking away until the barrier has worn off (while blink-microing weakened stalkers). Repeat. This is a detail of increasing the efficiency of trading stalkers with immortals that humans don't often even think about, let alone execute (because good blink control is often more impactful). That AlphaStar came up with this shows that it's not just about perfect execution of micro, but also perfect understanding of micro.

Re: AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

#330
post #46

This is really impressive, I didn't expect starcraft to be played this well by a machine learning based AI. I'm excited to read the paper when it comes out! That said, I'm not sure I agree that it was winning mainly due to better decision making. For context, I've been ranked in the top 0.1% of players and beaten pros in Starcraft 2, and also work as a machine learning engineer. The stalker micro in particular looked…

I am really impressed it learned when to pull probes in that game against Mana where the AI was pressured into his natural.

It was also extremely active with the stalkers, deciding to split them in three and not let Mana cross the map with his immortals.

Post reply on HN