Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grandmaster level in StarCraft II using multi-agent reinforcement learning

deepmind.com

Post reply on HN
Print view View original

291–300 of 321 posts

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#291

Post by hervature »

sanxiyn wrote: post #290
hervature wrote: post #253

Having a hard time to parse what is the action space here. The paper claims: AlphaStar’s action space is defined as a set of functions with typed arguments Looking at citation 7, it seems like they are structuring the action space as (First pick high level action)->(Pick argument 1 for action)->...->(Pick argument n for action). If this is the case, this seems to be "cheating" calling this AI as humans have completel…

It is extremely different from AlphaZero... In fact, they heavily rely on human knowledge, which is like opposite of AlphaZero. To quote the paper, "We found our use of human data to be critical in achieving good performance with reinforcement learning".

Ok, you’re right. I should’ve said AlphaGo. But that in itself shows what I mean that this is almost a step backwards.

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#292

Post by sanxiyn »

readparryrepost wrote: post #134

Love the work done into this, SC2 is such a 'human' game I'm impressed that it can be so successful but I'm wondering whether the model takes the raw graphical information or whether it takes some kind of full map representation. Some kind of machine representation might feel like cheating just a little as it's not quite competing on the same playing field as a human player.

It does not use raw graphical information, information is machine readable.

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#293

Post by sanxiyn »

hervature wrote: post #291
sanxiyn wrote: post #290

Earlier quoted context omitted.

It is extremely different from AlphaZero... In fact, they heavily rely on human knowledge, which is like opposite of AlphaZero. To quote the paper, "We found our use of human data to be critical in achieving good performance with reinforcement learning".

Ok, you’re right. I should’ve said AlphaGo. But that in itself shows what I mean that this is almost a step backwards.

AlphaZero was miraculously good, almost to the point of straining credibility. AlphaGo and AlphaStar are more like normal advances. They are mostly engineering, although theoretical contributions are not trivial. (Using reinforcement learning for value network in case of AlphaGo, and multi-agent self-play setup in case of AlphaStar, since straight self-play doesn't work.)

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#295

Post by drivebycomment »

PeterisP wrote: post #104
hoseja wrote: post #15

There has always been the issue of interface when playing videogames AI vs human. Either give the human a brain-computer interface or give the AI a mouse, keyboard, monitor, robot hands and a camera. Anything else seems inherently unfair.

The competition is not the goal - the point isn't to see "who's better", the competition with real humans is just a tool to help improve the reasoning, planning and decision making capabilities in a less artificial environment than usually. The purpose of that project is to use Starcraft as a playground to test, review and improve reinforcement learning methods so that they can be used for other needs. Any aspect of…

Yeah. Bulk of the other comments are useless for that reason. It's almost like humans have inferiority complex against AI...

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#296

Post by confidantlake »

DuskStar wrote: post #230
JamesBarney wrote: post #202

Earlier quoted context omitted.

There is a lot of strategy in RTS's but alpha star doesn't need to exploit any of it because it can win on mechanics. It has perfect macro, and perfect micro, which allows it to beat players without having to learn strategies or tactics. It's kind of like competing against a gorilla in boxing chess. Just because a gorilla is dominant doesn't mean boxing chess doesn't require chess skills, only that a gorilla doesn't…

Are you sure it has perfect micro? Can you even execute perfect micro on 30 actions per 5 seconds?

"Perfect" no, much better than any human yes. 30 actions per 5 seconds is 360 actions per minute. The very top players might have similar stats, but the similarities would end there. A lot of human actions are mindless spam clicking and those 300+ actions would contain several misclicks. The ai would never misclick. They can also do things like pull back every weakened unit before it died much more accurately than a human. A human knows to do these things but it has much less mechanical skill than the ai.

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#297

Post by Mirioron »

freddie_mercury wrote: post #285
thrower123 wrote: post #283

Earlier quoted context omitted.

Steam numbers are massively inflated by the tons of game purchases that come through sales, bundles, and giveaways and are never played. I have something like 600 games in my Steam library, and I would bet that the median playtime would be 0.0 hours... Just today I saw something that looked interesting, and it was 85% off because of a Halloween sale. So I grabbed it for a couple bucks. Free time being such a rare com…

Isn't that just proving my point? Most people who buy your game aren't going to invest even 10 hours in it or come close to maxing out the AI because they only picked it up because it was 85% off during a Steam sale. Maybe you'll be able to figure out from Early Access numbers how popular your game is and whether you need to invest in super-awesome AI. But it seems like for most games it would be a waste of effort. M…

>Maybe you'll be able to figure out from Early Access numbers how popular your game is and whether you need to invest in super-awesome AI. But it seems like for most games it would be a waste of effort.

Maybe the reason people quit your game is because of the AI. What made me stop playing Heroes of Might and Magic 6, Civilization 6, and Total War Three Kingdoms is the poor AI. It takes the fun out of the game, because at lower difficulties the AI puts up no challenge and at higher difficulties you're just exploiting the AI the same way over and over again to keep up.

You might think that that's fine, because you already got the money, but then you'd be looking at it from the perspective of singular games. But studios usually don't just release one game and then disappear. After my experience with HoMM 6, I had zero interest in HoMM 7. 2K Games could release Civilization 7 tomorrow, but without them showing that the game's AI is much better than in civ6, I probably wouldn't pick it up. Yes, the Civilization series will still have players, but it could easily just fizzle out like many other franchises and even genres.

You also can't add this in as a DLC, because people make up their mind about a game near the start. This only works if the game already has longevity, but at that point the game is already popular, so does a better AI at that point make a difference?

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#298

Post by Mirioron »

freddie_mercury wrote: post #279
the_af wrote: post #241

Earlier quoted context omitted.

I think part of the problem is that every videogame is a parlor trick in a sense. Unless you are a tournament player or someone who enjoys a really hard challenge, the bulk of the customers who buy the game just want to enjoy the illusion of a challenge. You don't really need a complex AI directing the hordes of enemies you want to defeat; you just want the illusion of battling enemies and emerging the victor, and ho…

To extend your point, it is also clear from analytics in Steam & mobile gaming that games have massive amounts of player dropout. It is pretty rare, even in relatively short single player campaigns, to see 30% of people who bought the game finish it. So effort making the AI even better is always going to only be for an extremely small niche of players that play a game long enough to exhaust the limits of simple AI. J…

It doesn't matter how short or long the singleplayer campaign is. AI in games is so simple and rudimentary that you'll have it figured out in most games in less than an hour. You might not figure out the most effective ways to exploit it, but usually you'll figure out something. This alone can make the experience boring, especially when your game is heavily focused on playing against the AI.

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#299

Post by Mirioron »

the_af wrote: post #241
Mirioron wrote: post #214

Earlier quoted context omitted.

> There is a genuine advance here, but keep in mind that when an AI is developed by the game developers, they're not necessarily playing to win, but to make the AI fun to beat, and without using too much computer power, which would make the game slower. This point is being brought up a lot, but I don't really buy it. Yes, there have been instances where the AI being too good discouraged players from playing the game…

I think part of the problem is that every videogame is a parlor trick in a sense. Unless you are a tournament player or someone who enjoys a really hard challenge, the bulk of the customers who buy the game just want to enjoy the illusion of a challenge. You don't really need a complex AI directing the hordes of enemies you want to defeat; you just want the illusion of battling enemies and emerging the victor, and ho…

>You don't really need a complex AI directing the hordes of enemies you want to defeat; you just want the illusion of battling enemies and emerging the victor, and how the game accomplishes this is not that relevant...

Yes, and the current game AIs are completely and utterly incompetent at it. AI in video games only provides a challenge when the mechanics of the game are challenging or the AI is cheating.

>Do we really need something more complex (and expensive!) when killing zombies in L4D?

Not in L4D, because you don't play the game to kill zombies. You play the game to play with other people. If L4D didn't have coop then the game would likely not have gained popularity.

You're right that players want a challenge that they can overcome, but current AI in video games does not offer that. The AI always ends up being so weak that play against the AI cannot be the focus of the game. The focus has to be in mechanical play, puzzles, story, atmosphere, multiplayer or something else. The only way AI keeps up is with inhuman mechanical play or cheating. Neither feels fun to play against, because the counter to it is to figure out how to exploit the AI, after which the game becomes more or less trivial.

Re: Grandmaster level in StarCraft II using multi-agent reinforcement learning

#300

Post by narrator »

sanxiyn wrote: post #287
narrator wrote: post #162

We need to create an open source organization to build these brains at home because: "Recomputing the AlphaGo Zero weights will take about 1700 years on commodity hardware."[1] I see these huge AI brains creating a new class divide between people who have access to these new AI brains and those who don't. The mission of open source has always been to break down these barriers to empowerment with technology. Thus, thi…

AlphaGo Zero was extremely computation inefficient because in (IMO misguided) attempt to be "general", DeepMind avoided even slightly specializing training to Go. KataGo optimized training to Go, slightly, and obtained 100x speedup, and that was low hanging fruit.

1700 years sped up 100x is still 17 years though. Even so, for the models that actually need it, Google and other people with huge GPU farms will have an enormous advantage over a single individual and only a large community can have any hope of challenging their ability to dominate.
Post reply on HN