OpenAI at the Dota 2 World Championships • Hacker News Board

Re: OpenAI at the Dota 2 World Championships

#51

Post by danielmorozoff » Sat, Aug 12, 2017, 12:40 AM UTC

Zyst wrote: post #43

kibwen wrote: post #42

In the interest of a "fairer" comparison, I wonder how much of a difference it would make to force the AI to simulate mouse/keyboard input and interpret the raw screen buffer output, rather than using direct APIs into the game's guts, to more faithfully emulate its human opponent. I'm guessing the peripheral inputs wouldn't be much of a hurdle, but the image processing step could be very interesting.

Raw screen is not as big of a deal as it would seem, in DotA and Starcraft you can glimpse most of the information you can see on a screen through the minimap.

It would seem to me that this is an oversimplification. Doing screen based video processing is not simple and is not exact especially where camera movements are independent of game dynamics as well as the difficulties inherent in doing fast frame classification- there has been some good work in this direction recently but far from perfect(yolo, ssd). The route deep mind took in partnering with the game manufacturers to gain inputs directly while receiving global information about game state seems the simplest for their goal of training rl agents. We considered doing a project around this but figured it was a massive iceberg.

Re: OpenAI at the Dota 2 World Championships

#52

Post by UnpossibleJim » Sat, Aug 12, 2017, 12:41 AM UTC

These types of MOBA games are a good precursor of military unit management. To be honest, I'm not totally sure whether to be happy or sad about this sort of thing. The strategic control of drone units on a combat field, without the loss of personel (on the countries with this technology, anyways) should make me happy, and does, to a point. BUT the potential for abuse (and, no, I haven't even begun to extrapolate towards the Terminator, nightmare scenarios) is rampant. Fewer people with a conscience on the battlefield or controlling the apparatuses of war may or may not be worth the cost of the lives of young men and women..... but I think wars should be fought by old men and women with swords, anyway. If you're old and can look i to the face of your enemy while you kill them, there's a better chance it's worth killing and dying for (by the numbers, anyways).... plus, the President, Congress and the Senate have to serve in combat positions, in my little fantasy scenerio =)

Re: OpenAI at the Dota 2 World Championships

#53

Post by nether » Sat, Aug 12, 2017, 12:42 AM UTC

No!

Re: OpenAI at the Dota 2 World Championships

#54

Post by ahh » Sat, Aug 12, 2017, 12:42 AM UTC

The developer interviewed claimed that there was no domain-specific knowledge and implied (though didn't explicitly state) there wasn't any training against non-OpenAI bots or players. (I'd love to know the reward function they used for whatever Q-learning variation they ran with.)

If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking. (I can't find a good GIF, but this is walking in a wiggly path in front of the first wave of neutrals on your side, delaying their progress and pushing the lane towards you, which is good for ~reasons.)

Creep blocking isn't all that hard in dexterity--I am a terrible dota player and I can more or less do it. And it's one of the most common pieces of dota knowledge; every pro player does it and since it's relatively easy compared to a lot of pro micro, everyone else rapidly learns they should.

But nevertheless--the bot had enough games that it could randomly jump in front of the wave enough times that it noticed a win rate improvement for that slight wave push, and begin to do it intentionally? (And then get good at it?) Damn.

One thing I don't really know about Q-learning and the typical nets used for it: I am guessing it is likely that internally to the bot's evaluation functions, there is some learned feature whose activation correlates well to the location of the wave equilibrium (since that's a feature that correlates well with winning!) At that point, is it likely that the bot can learn in smaller increments--that is, it knows that pushing equilibrium towards itself is good, and thus randomly creep blocking a little becomes reinforced (rather than having to notice the creep block's effect on game wins?)

Re: OpenAI at the Dota 2 World Championships

#55

Post by zamalek » Sat, Aug 12, 2017, 12:44 AM UTC

mAritz wrote: post #18

sondr3 wrote: post #11

Wow, that was not even close either. And it only takes it two weeks to reach that level of play? OpenAI should create a team of five (as they said they would) and let it play in the qualifiers next year.

To be fair, the 1v1 Shadow Fiend matchup with these rules is A LOT about exact distances and timings. To which the bot has direct API access and the human has to got the whole GPU -> screen -> eyes -> brain -> muscles -> mouse -> CPU cycle.

Shadow Fiend really comes into its own when blind shots happen, as with many other heroes, fights can be lost to line-of-sight. That requires knowing your opponent's mind and prediction. I can't watch the video just yet, but this is the human behavior you should look out for (such as AlphaGo demonstrated repeatedly in the most recent match).

Re: OpenAI at the Dota 2 World Championships

#56

Post by needz » Sat, Aug 12, 2017, 12:48 AM UTC

deivid wrote: post #44

ronald_raygun wrote: post #33

Earlier quoted context omitted.

I'm not sure about DOTA, but in LOL there is also champion select which happens before the game starts, where you try to pick/ban/steal champions to get a good team comp and get the other team to have a bad team comp. It's lots of know overall strategy, what your enemy likes, and some game theory. You can lose the game before your summoner even hits the field.

Main difference is all Dota heroes are viable in competitive. Also, there's no mirror match

Only 3 heroes not picked in TI, iirc. It hasn't always been this diverse though. This patch is particularly well balanced.

Re: OpenAI at the Dota 2 World Championships

#57

Post by jsmthrowaway » Sat, Aug 12, 2017, 12:50 AM UTC

hhmc wrote: post #41

mAritz wrote: post #39

Earlier quoted context omitted.

Why? Everything it needed for that 1v1 was available. There was nothing in the FoW that needed to be known, it can keep timers about enemy cooldowns, gold and xp (approximations at least since it's somewhat random).

I'm not suggesting they had information that a human player wouldn't (i.e. cheating), I just suspect that they may have access to a richer api.

Which would give them what? The second clause of your sentence contradicts the first.

Re: OpenAI at the Dota 2 World Championships

#58

Post by Tangokat » Sat, Aug 12, 2017, 12:51 AM UTC

It is so weird reading all these comments. Almost half of them start out as "it's impressive.. but". Is it human nature or are HN commenters just so sceptical/negative of all the new tech?

One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery. Surgery is not "just" run on a computer it has a physical component too. Is that really the next step or was he just throwing out a random example people could understand?

Re: OpenAI at the Dota 2 World Championships

#59

Post by habitue » Sat, Aug 12, 2017, 12:52 AM UTC

So an interesting thing you could do as a game company with an "unbeatable" bot is to use it to balance a metagame. Let the bots all play each other and tweak character stats etc until they win a proportional amount of the time. (This presupposes that the bot learns the game the way OpenAI claims it is, without needing to learn from player replays or from playing in a very different way than human players do with tree search etc)

Re: OpenAI at the Dota 2 World Championships

#60

Post by darrenkopp » Sat, Aug 12, 2017, 12:52 AM UTC

Really was quite interesting to watch (https://www.twitch.tv/videos/166172514?t=7h3m10s). Honestly, the bot played _extremely_ well, but I think the biggest advantage was how much faster it's reaction time was and it's movements were likely much more precise than a human is with a mouse.

I'm pretty interested in seeing their 5v5 results as well. It seems like that will have similar results as the bots can coordinate, but it's still a bit up in the air.

I'm really not sure how well a bot like this would do with 4 human teammates though. I would guess the bot would be a strong laner, but fairly weak overall due to it's inability to communicate, though if it's really good at learning how it's current teammates are playing as it goes, it may do alright.

I'm also pretty curious about the match limitations: no shrines (regen), no soul ring (mana regen at expense of some health), no raindrops (fixed amount of magic damage block)