Grandmaster level in StarCraft II using multi-agent reinforcement learning

Starcraft is a relatively simple game (which is one of the main reasons for its enormous popularity), so seeing an AI do well at it is not particularly impressive.

I'd have been far more impressed had an AI beaten some of the best wargamers at a complex wargame, or had it beaten some of the best text-adventure game players at novel text adventure games neither of them had played before.

The former would be difficult because of the enormous search space that results from the large variety of units, possible actions, and board size in complex wargames, and the latter is from the creativity required to do well at unfamiliar text-adventure games (which are hugely varied and can't be boiled down to any simple set of rules).[1]

Text adventure games would present a further difficulty in that not only do they not have very many players compared to games like chess, go, or Starcraft, but their players aren't normally ranked relative to one another, and there's no enormous record of human-played games to draw on. Further, in text adventures sometimes there are clear winning conditions and sometimes not, and sometimes clear signs of progress and sometimes not. It's also hard to objectively rate a text adventure game's difficulty. Without these features researchers would have a much harder time training an AI to do well against humans.

Another game which I briefly thought about nominating as a challenge for AI's was Factorio, due to its enormous complexity. However, when I thought about it for a little while I realized that this was a game that computers would find it easy to beat humans because they'd do a much better and faster job at number crunching and optimizing. Factorio is really more of a game where a human almost tries to emulate a computer, much like programmers do when they're mentally evaluating or stepping through an algorithm or trying to optimize something.

Of course, seeing an AI beat humans at any of these games would still make the front page of HN, but most gamers (nevermind most non-gamers) who would probably just say "What?" ... and that's probably yet another reason for the reasearchers choosing Starcraft. It's relatively easy, low-hanging publicity fruit, where they could claim success and a relatively large number of people would kind of understand why it was an accomplishment.

[1] - Yes, I'm aware of the recent HN post about fuzzing done on the Z-machines, but by the author's own admission that fuzzer cheated by having access to the game's word dictionary, and it also cheated by having access to the game's internal state -- neither of which a human player would have access to. Also, many Z-machine games are ancient and relatively simple and straightforward compared to more modern games.

Having played starcraft - the idea doesn't work. Starcraft happens to quickly, with too much precision, and too low a tolerance for latency to wait for a human to absorb understand and execute instructions.

Having programmed simple NNs the idea doesn't work. The amount of time you'd have to spend having humans execute instructions during training would be astronomical. They were training 16000 games simultaneously, for 44 days, most likely running at some multiple of how fast the game normally runs. Moreover you lose out on supervised learning, because we don't data sets of "the human told the other human to do this", we only have data sets (almost a million games large) of "the human did this".

You only need excess dexterity than your opponent, and only for a couple seconds, to decisively win a match in SC2.

Error rates, EPM and APM are all red herrings.

I see a lot of comments downplaying this achievement, saying it's not impressive or it won't be impressive until X condition is met.

I welcome skepticism and criticism for this sort of thing, and think most of it that I've seen here is well founded. But I would like to take a second to explain why I think this, and really all the progress in this area, is actually a really impressive achievement to me.

Let me try and frame this from the computers perspective. Let's assume a resolution of 1024x780. I'm not sure what size frames they actually feed their agent but it's not that important to the discussion, the point is it's a big image, and according to the article this agent is learning from pixels. So, you the computer are given let's say 1024*780 = 798720 numbers to look at. You then choose a number between 0 and 798720 (or the crazy 10^26 number the article gives as the possible number of actions at each frame) as your action for that frame, and then you get another 798720 numbers to look at. After the round is over (on average 20 minutes, if you make a decision every frame that's 20x60x60 = 72000 rounds). You get one number telling you how well you did. You repeat the process and get a new number. It's higher this time! But what is the cause? was it that click you made on frame 22456? or maybe that unlikely move you made on frame 4567?

Obviously I'm oversimplifying here, and the numbers are probably wrong. but I still think what I've said gives the right idea for what kind of task we (as a society/community/whatever) have somehow gotten a computer to solve. Computers are DUMB, the fact that it's able to play this game at all, let alone at a high level, is still a minor miracle to me.

It would be interesting to see if there are any meta reports on people doing research on the topic of resource constrained AI. Has anyone explored whether or not an AI can improve on its own algorithm while being heavily resource constrained?

Is the ability to use less resources a limit of our current hardware? Do we see the current hardware performance improvement trajectory being able to reduce the amount of resources an AI consumes to perform a task?

Are our algorithms simply not tuned well for using smaller resources? Could we build better algorithms around resource deficient environments?