Live data from Hacker News

Building AI that can master complex cooperative games with hidden information

ai.facebook.com

21–29 of 29 posts

Re: Building AI that can master complex cooperative games with hidden information

#21

Hi! I'm one of the authors on the paper. We'd be happy to answer any questions. Ask us anything!

Damn, I haven’t gotten around to fully reading Pluribus yet, and now there’s more? Congrats on the results! What’s next?

Thanks! We're looking in a few different directions, but one thing I'm excited about is mixed cooperative/competitive settings. In poker, there is no room for cooperation. In Hanabi, you are 100% cooperating with your teammates. But most real-world situations, like a negotiation, are somewhere in between. The AI techniques for these settings are not too strong yet.

Re: Building AI that can master complex cooperative games with hidden information

#22

Hi! I'm one of the authors on the paper. We'd be happy to answer any questions. Ask us anything!

Isn't this not the same thing as AI that can beat humans at things like Bridge where the bidding game matters quite a lot? IIRC in Hanabi the fact that there is imperfect information does not really matter that much for strategy, where as in things like League of Legends or Bridge or many of those types of games it really does matter quite a lot.

Bridge has a similar challenge, though from what I understand Bridge AIs are not superhuman yet. I suspect our techniques could be applied to Bridge, though they may need to be adapted a bit.

The imperfect information in Hanabi absolutely matters a ton. It's not an interesting game without it.

Re: Building AI that can master complex cooperative games with hidden information

#23
post #10

If you're wondering why this is interesting, games that AIs excel at like chess/checkers/go are all two-player, zero-sum, perfect-information (everyone knows everything), deterministic games, so you can exactly predict your opponents behavior by simulating "what would I do if I were them, and trying to make me lose". The only real hard problem in this space is extreme branching factors. Everything gets vastly more co…

I just struggle to see the real challenge in this argument. I can see that simulating something more complex than a Go game is a real engineering challenge - simulating a Go game from start to finish is really easy apart from assigning a value function to each move.

That being said:

> hidden information forces you to manage how much you reveal to your opponent

It is hard to see how that could be harder for a learning AI to pick up than any other game action with long-term benefits. It might reveal that AI are so much better at humans in games like Go and Chess that they aren't even exploiting long-term links between cause and effects, but I suspect from the example we've seen with Go that this is a solved problem.

> and requires you to simulate multiple "alternate futures" based on things you learn after making a decision,

That is describing a tree search. Why is that theoretically difficult? I can see it is a real engineering challenge to simulate a complex environment.

> and randomness is equivalent to an extra player that makes irrational unpredictable moves.

Computers handle randomness much better than humans at all levels because they have an actual statistical grounding. They still aren't very good at it, but they sure thrash humans.

I suppose basically I can see why a hidden-information game could be intractable because simulating the environment is so hard that it can't be done and breakthroughs in simulation must be found to train the AI. But I don't see why that is being linked to hidden information and cooperative gameplay. We know humans do fine without utilising hidden information, because they can play these games and they don't know any hidden information.

Re: Building AI that can master complex cooperative games with hidden information

#24
post #10

If you're wondering why this is interesting, games that AIs excel at like chess/checkers/go are all two-player, zero-sum, perfect-information (everyone knows everything), deterministic games, so you can exactly predict your opponents behavior by simulating "what would I do if I were them, and trying to make me lose". The only real hard problem in this space is extreme branching factors. Everything gets vastly more co…

In particular, communication is in some ways a complex multiplayer game requiring multi-level modeling of other participants. In the ideal case, it can be modeled as a cooperative game. In suboptimal cases that sadly occur often in the real world, it's a game where you're fractionally cooperating and fractionally competing with every other player, and the degree to which you're cooperating or competing with any given…

Unfortunately Facebook’s approach sidesteps this complexity by ensuring each player uses the same random seed and searches policy based on information they can all see. It’s not really solving the problem as intended in my opinion.

Re: Building AI that can master complex cooperative games with hidden information

#25

Earlier quoted context omitted.

What's FB going to do with this?

Open source it, learn from it, and build upon it to continue to push forward the frontier of AI.

That would be nice. What is Facebook AIs take on ethical use of its research?

Facebook probably pays through the nose for AI research and probably wants a ROI. Facebook makes money by building better user models and spamming targetted ads. Some of them are getting scarily good.

Re: Building AI that can master complex cooperative games with hidden information

#26
post #2

they (basically) applied the ideas from a bot that plays poker to another game. it's interesting work, though perhaps not groundbreaking. This idea of selfplay + counterfactual regret minimization does seem to be the superior way to solve game theoretic problems. Identifying valuable game theoretic problems remains a challenge...

The search algorithm shares a lot in common with our Pluribus poker AI ( https://ai.facebook.com/blog/pluribus-first-ai-to-beat-pros-... ), but we added "retrospective belief updates" which makes it way more scalable. We also didn't use counterfactual regret minimization (CFR) because in cooperative games you want to be as predictable as possible, whereas CFR helps make you unpredictable in a balanced way (useful in…

I just spent three weeks going through your research. Thank you for that work, especially the supplementary materials.I wish I'd known how much the ideas in the pluribus paper depended on reading the libratus paper.

I see what you're saying about the real time search (which took me quite some time to understand). I came up with a way to do that from disk due to memory limitations. It limits the number of search iterations but doesn't seem to have a huge negative impact on quality so far.

Anyway, thanks again!

Re: Building AI that can master complex cooperative games with hidden information

#27

They should tackle Starcraft II next like DeepMind has with AlphaStar, or at least a similar real time RTS like Starcraft with a fog of war and a partially observable state.

There are unique challenges around learning effective communication protocols that appear in cooperative settings, which was the focus of this work. Getting robust superhuman performance in SC2 remains an interesting challenge, though.

Re: Building AI that can master complex cooperative games with hidden information

#28

Earlier quoted context omitted.

Hanabi definitely has traits involving theory of mind that I've not seen present in other games. For example, I've played Hanabi with 3+ players where the person before me deliberately gave a misleading hint to the person after me. For example, "this is your only blue card" indicating a blue 5 even though only blue 1-3 have been played. They were counting on me to anticipate that the mislead person was now very likel…

> For example, I've played Hanabi with 3+ players where the person before me deliberately gave a misleading hint to the person after me. For example, "this is your only blue card" indicating a blue 5 even though only blue 1-3 have been played. They were counting on me to anticipate that the mislead person was now very likely to waste that valuable card & realize that I could only reasonably avert that misplay by play…

We haven't yet analyzed the gameplay to look for examples of these well-known human Hanabi conventions. All the code and agents are open-sourced though, so feel free to take a look!

Re: Building AI that can master complex cooperative games with hidden information

#29

Earlier quoted context omitted.

In particular, communication is in some ways a complex multiplayer game requiring multi-level modeling of other participants. In the ideal case, it can be modeled as a cooperative game. In suboptimal cases that sadly occur often in the real world, it's a game where you're fractionally cooperating and fractionally competing with every other player, and the degree to which you're cooperating or competing with any given…

Unfortunately Facebook’s approach sidesteps this complexity by ensuring each player uses the same random seed and searches policy based on information they can all see. It’s not really solving the problem as intended in my opinion.

We are entirely focused on the self-play setting in which the goal is to learn the highest performing policy for a team of agents all trained together. The Hanabi Challenge also outlines an ad-hoc setting in which you need to adjust to the diverse policies of other agents in the team on the fly.
Post reply on HN