Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning Machine Learning for Game Development

ai.googleblog.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Learning Machine Learning for Game Development

#61

Post by ericjang »

CJefferson wrote: post #38
ericjang wrote: post #10

Earlier quoted context omitted.

Interesting! Is there some kind of meta-objective for "AI optimal" that could have replaced the 6 or 7 iterations you did with human R&D? For instance, if you had real human playtesters interacting with the prototypes, is there some signal you could extract to measure that it's "good"?

The problem is AIs are very good at optimising what you asked them for, rather than what you meant to ask for, and figuring out what you want is super hard :) As a simple example: * Start by optimising "players can always do something on their turn" -- but that just ends up with everyone always having exactly one thing they can do (no choice). * So then say "give players more things to do each turn" -- but then they…

Got it. I do research in reinforcement learning and I can sympathize with the difficulty here - in my experience, even something as simple as "I want to balance two objectives: 1) the agent should get a high score and 2) the agent should try to make as few decisions as possible" tends to result in the agent doing neither of those things well"

Re: Learning Machine Learning for Game Development

#62

Post by [deleted] »

ZephyrBlu wrote: post #60
anonymoushn wrote: post #58

Earlier quoted context omitted.

> Most games have enormous action spaces, so you can't easily compute a probability. Saying you can't compute something by hand on a single sheet of paper is different from saying it doesn't exist. In Chess, your win probability is either 0 or 1 (you either can force a win or you cannot), but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1.

> but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1 My understanding is that Chess engines do not evaluate a 0-1 value to representing chance to win. They evaluate the perceived material advantage from the current board position. See: https://www.chess.com/blog/ChessMN16/how-to-read-engine-eval...

[deleted]

Re: Learning Machine Learning for Game Development

#63

Post by ZephyrBlu »

anonymoushn wrote: post #59
ZephyrBlu wrote: post #57

Earlier quoted context omitted.

Context of the game = previous events that occurred. Don't you think it's a bit odd to be forecasting a "win probability" value without taking the events of the game into account? > why can't it be included in the encoding of the game state? How do you encode time series or event data into singular values?

> Don't you think it's a bit odd to be forecasting a "win probability" value without taking the events of the game into account? No. To the extent that these events have no impact on the rules or the outcome of the game beyond the current game state, they don't influence how much equity a player has in the game. You may be interested in "The probability that I win this game given my beliefs about the tendencies of my…

> To the extent that these events have no impact on the rules or the outcome of the game beyond the current game state, they don't influence how much equity a player has in the game

This is a really odd statement. Past events can most definitely affect events beyond the current game state.

I think the AlphaStar paper [0] has a lot of good examples of my overall points. Here are a few excerpts:

"Central to AlphaStar is a policy [formula], represented by a neural network with parameters θ that receives all observations [formula] from the start of the game as inputs, and selects actions as outputs"

"To manage the structured, combinatorial action space, the agent uses an auto-regressive policy7,10,11 and recurrent pointer network"

I.e. the policy does account for past actions, and these are not encoded in any way.

"The agent parameters were subsequently trained by a reinforcement learning algorithm that is designed to maximize the win rate (that is, compute a best response) against a mixture of opponents"

I.e. the winrate value is _not_ a "win probability", it's a maximization like I said in my original comment.

Do these hold true for discrete action games and/or perfect information games? I don't know, but I find it likely they hold for at least discrete, imperfect information games.

[0] https://www.nature.com/articles/s41586-019-1724-z.epdf?autho...

Re: Learning Machine Learning for Game Development

#64

Post by juanuys »

adkadskhj wrote: post #11
CJefferson wrote: post #3

I'm in this research area, it's facinating, but also (like many things in AI) easy to get wrong. The trickiest bit is deciding exactly what you want your AI to optimise for -- If you want a game where "anyone can win", you end up with coin toss. If you want a game where "skill is everything", you can accidentally end up with "who can click fastest". On a project I recently worked on (and I'm writing up), we ended up…

Personally, i'd like to see AI used for "simple" things, not often tied to the core gameplay loop itself. Ie conceptually if you could tie the right gameplay components to an RNG function then the user could get new/unexpected experiences for a longer time. The problem is RNG often makes crap outputs. I'd be curious to see ML take this problem. I imagine the examples are limitless, but one that comes to mind might be…

> Procedurally generated worlds often have that feeling. Things don't make sense, etc

Spelunky randomly stitches together pre-authored rooms (more like room patterns, each with a sprinkling of RNG) to get around this.

https://bossfightbooks.com/products/spelunky-by-derek-yu

Re: Learning Machine Learning for Game Development

#65

Post by TylerLives »

ZephyrBlu wrote: post #60
anonymoushn wrote: post #58

Earlier quoted context omitted.

> Most games have enormous action spaces, so you can't easily compute a probability. Saying you can't compute something by hand on a single sheet of paper is different from saying it doesn't exist. In Chess, your win probability is either 0 or 1 (you either can force a win or you cannot), but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1.

> but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1 My understanding is that Chess engines do not evaluate a 0-1 value to representing chance to win. They evaluate the perceived material advantage from the current board position. See: https://www.chess.com/blog/ChessMN16/how-to-read-engine-eval...

Why do you think it's unreasonable to interpret the value function as probability? I think chess engines don't do it because of draws, but you could look at a chess position and estimate what are your chances of winning/drawing/losing.

Re: Learning Machine Learning for Game Development

#66

Post by ZephyrBlu »

TylerLives wrote: post #65
ZephyrBlu wrote: post #60

Earlier quoted context omitted.

> but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1 My understanding is that Chess engines do not evaluate a 0-1 value to representing chance to win. They evaluate the perceived material advantage from the current board position. See: https://www.chess.com/blog/ChessMN16/how-to-read-engine-eval...

Why do you think it's unreasonable to interpret the value function as probability? I think chess engines don't do it because of draws, but you could look at a chess position and estimate what are your chances of winning/drawing/losing.

If the probability of winning was static I would agree that the evaluation function is (Or can be) a prediction of win probability.

However, because the probability of winning is clearly dynamic over time, trying to predict that probability is inherently a future prediction of the outcome (I.e. a forecast) rather than the prediction of a value.

In that case, it does not make sense to view the evaluation function as probability because a forecast is not the same thing as a probability.

I'm looking at this from the perspective of the game. From the perspective of any given state in isolation I agree that it makes perfect sense, but in the context of a game it doesn't.

This is why I think that plotting all these values on a chart together and calling it "win probability" is non-sensical. It does give you an idea of the game state, but it's not a probability.

Re: Learning Machine Learning for Game Development

#67

Post by sdwr »

ZephyrBlu wrote: post #44

I have a bone to pick with these "win probability" charts. If it was truly a "win probability" chart, that means it's a forecast. Except it's a shit forecast because you're trying to predict really freaking far into the future (You don't even know how far because the game could end at any point in time). It also makes zero sense. Think about it, I say "you have a 30% chance of winning the game from this position". Wh…

Like any metric, it's not perfect, but there's a lot of good information there.

Dota added a win prediction graph a while back that's been valuable to look at. It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting. For example, it takes character selection into account, and the initial prediction can get to 60/40 before the game even starts.

The bit that I found the most interesting is when the prediction doesn't line up with my intuition of the game state. I've had a couple of comebacks from a predicted 1% win chance, and all those felt more like 20-30% comebacks, where we were down but definitely not out. A map is not the territory, and that's easiest to see when you fall off the edge of the map.

Re: Learning Machine Learning for Game Development

#68

Post by ZephyrBlu »

sdwr wrote: post #67
ZephyrBlu wrote: post #44

I have a bone to pick with these "win probability" charts. If it was truly a "win probability" chart, that means it's a forecast. Except it's a shit forecast because you're trying to predict really freaking far into the future (You don't even know how far because the game could end at any point in time). It also makes zero sense. Think about it, I say "you have a 30% chance of winning the game from this position". Wh…

Like any metric, it's not perfect, but there's a lot of good information there. Dota added a win prediction graph a while back that's been valuable to look at. It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting. For example, it takes character selection into account, and the initial prediction can get to 60/40 before the game even starts. The bit that I…

I'm aware of Dota's win prediction chart.

Do you know about STRATZ? They also have a win probability chart on their website as well as predictions for each hero in the game.

In their Discord, different people have asked the same questions about these "win probability" values multiple times because often they're very unintuitive (Because they are not actually probabilities).

> It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting

If it wasn't a black box you would know why it's weighted at a particular value.

I think it's mostly a curiosity and not that helpful if you're trying to analyze your games.

Re: Learning Machine Learning for Game Development

#69

Post by anonymoushn »

ZephyrBlu wrote: post #63
anonymoushn wrote: post #59

Earlier quoted context omitted.

> Don't you think it's a bit odd to be forecasting a "win probability" value without taking the events of the game into account? No. To the extent that these events have no impact on the rules or the outcome of the game beyond the current game state, they don't influence how much equity a player has in the game. You may be interested in "The probability that I win this game given my beliefs about the tendencies of my…

> To the extent that these events have no impact on the rules or the outcome of the game beyond the current game state, they don't influence how much equity a player has in the game This is a really odd statement. Past events can most definitely affect events beyond the current game state. I think the AlphaStar paper [0] has a lot of good examples of my overall points. Here are a few excerpts: "Central to AlphaStar i…

> I.e. the policy does account for past actions, and these are not encoded in any way.

Alright. They should be then.

I misunderstood your objection and I would express a part of it as "the game state is not fully encoded." As an example, my encoding of Splendor gamestates includes which card is secretly in an opponent's hand if the opponent reserved it when it was face-up, and I regard this as "part of the game state," and it seems like the fine article's encoding lacks such information.

On the other hand, this discussion has made me realize my Splendor game state encoding is also lossy compared to the information needed for actual play by experts. If an expert player has 2 red chips, then reserves a face-down rank 3 holding, then takes 2 green chips the next turn, that's very different from if they started with the green chips and began collecting red chips after seeing their secret high-point-value card. My encoding does not account for this and I am struggling to think of how to fix it.

Post reply on HN