Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goldman Sachs model to predict World Cup game results didn’t come close

bloomberg.com

Post reply on HN
Print view View original

121–130 of 136 posts

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#121

Post by thaumasiotes »

dasil003 wrote: post #119
PaulRobinson wrote: post #76

Earlier quoted context omitted.

> Brazil had 27 shots and 9 on target with 59% posession. Belgium only had three shots on target, and made two of them to win. A modern model would accommodate for the fact that those numbers alone mean nothing, because they don't. Those are the numbers broadcasters reluctantly put on a screen for entertainment value, but they don't have real analytical power because they have no comparative metric. How up or down we…

Sure those are just basic stats and could be improved probably, but they do reflect the reality that Brazil should have won; they got unlucky with an own goal, and they made some key mistakes at critical times, failing to finish great chances. You're not going to find a statistical approach that will account for the subtleties that led to this outcome. The problem with soccer stats in general is that everything hinge…

> You're not going to find a statistical approach that will account for the subtleties that led to this outcome. The problem with soccer stats in general is that everything hinges on low-frequency events based on subtle differences of timing and space.

I think this deserves to be elaborated a bit: a game in which 1 is a good score, and often a game-winning score, is never going to be accurately predicted based on a statistical approach, because scoring is too rare for a statistical approach to work well. Low scores mean that individual games have an extremely large element of chance.

Imagine one team is about 4% better than another team; they should be favored about 51-49 to score a point. If a game scored 300 points, that difference would be perceptible within one game. But to resolve the same difference accurately in games that score 3 points each takes many, many, many games.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#122

Post by kgwgk »

thousandautumns wrote: post #112
kgwgk wrote: post #92

Earlier quoted context omitted.

If you expect to get it right (in this particular prediction, Clinton to win) with 95% probability, what does it mean to say that this 95% is with low confidence or with high confidence?

Not OP, but that opens a whole different can of worms. “Confidence” has a specific meaning in the context of statistical theory, and specifically in a particular flavor of statistics called “frequentism”. I won’t get into what is involved in frequentism, and how it differentiates itself from the alternative, Bayesianism, but essentially “confidence” refers to a measure that really says more about the statistical meth…

I agree that this is a different can of (nasty) worms.

The message I replied to said that > It was not wrong to say Hillary had a 95% chance of winning the presidential election,

Frequentist inference cannot be interpreted as a probability unless one goes through some (often misunderstood, as you pointed out) contortions. In your scenario where you have 95% confidence of something it would be wrong to say that Clinton had a 95% chance of winning.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#123

Post by cambalache »

PaulRobinson wrote: post #76
basch wrote: post #62

Earlier quoted context omitted.

Their model also had France at 2nd most likely, Belgium at 5th, and England at 7th. 3 of their top 7 made the Semi-Finals, and they called the eventual winner as Second Most Likely, and more likely than Germany. They actually predicted the Brazil/Belgium game in the Quarter Finals, but got the winner wrong. Brazil had 27 shots and 9 on target with 59% posession. Belgium only had three shots on target, and made two of…

> Brazil had 27 shots and 9 on target with 59% posession. Belgium only had three shots on target, and made two of them to win. A modern model would accommodate for the fact that those numbers alone mean nothing, because they don't. Those are the numbers broadcasters reluctantly put on a screen for entertainment value, but they don't have real analytical power because they have no comparative metric. How up or down we…

Instead they mean, a lot. Shots on target is the proxy you have (except of course goals) to derive who team dominated more. As a matter of fact if you follow the sport, you will know most coaches will be satisfied if the shots on target is good, even if one particular game no goals are scored. The tragic thing for Brazil is that the WC is a short and direct elimination tournament. A bad game and you are gone.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#124

Post by ghayes »

jtolmar wrote: post #120
thousandautumns wrote: post #104

Earlier quoted context omitted.

The point of the model is absolutely not to reduce uncertainty, it is to quantify it, which are two very different things. No model reduces uncertainty in a probabilistic sense. And no, you don’t need statistics or machine learning to say “there is a lot of uncertainty”, but you do in order to quantify that uncertainty.

Say I have two models - model A returns around 20% likelihood that the top team wins the world cup, and model B returns around 80% likelihood. I use both of the modeling techniques a few thousand times in various parallel universes, and both of them are exactly right - 20% of 20% predictions result in a win, and so on. Despite them both quantifying uncertainty accurately, isn't model B still better?

Think about the actual uncertainty since anything can happen in the game itself. The easiest way to look at this is to play the game 100 times in a row (preferably in parallel universes, as you say). If team A wins in 60% of games, then that caps the ability to predict the result. You can predict a die roll to be 6 with an certainty of 17%. You can’t do any better.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#125

Post by kgwgk »

LeifCarrotson wrote: post #81
kgwgk wrote: post #28

Earlier quoted context omitted.

> had a 95% chance [...] but the confidence was low So she had 95% chance of winning with 50% probability or what?

There are two probabilities in question: The first, of course, is the probability of victory. The second is the probability that the first probability is correct. Consider: If someone offered to give you $2 every time a fair coin toss came up heads, or take $0.50 every time it came up tails, you'd be foolish not to take that bet a million times as you can because you know that the coin has exactly a 50% chance of com…

There are two probabilites is you want to make it so in your model. In the coin example it may make sense, you model the coin as a binomial probability and you can estimate it. You can repeat the events, it makes sense to talk about frequencies and you can improve your estimate of the parameter.

In the election model it's not clear to me what's to gain by saying that there are two probabilities (or more) instead of one. There is one single event.

>Hillary had a 95% chance to win the election. But on top of the fact that 1 in 20 times she'd lose that election if that really was the probability,

Which is the only thing that matters if we say that the probability was 95%.

> the 95% number was uncertain because the measurements were difficult to pin down - maybe she'd have lost 1 in 40 times, or maybe she'd have lost 1 in 5 times.

You have lost me here. Did she have a 95% chance to win or not?

If this 95% is uncertain, because it could have been 97.5% or 80%, then the probability would be the weighted average of those numbers and not 95%. And if it was so uncertain that nothing was known at all it would be 50%.

Consider the following cases:

a) You are going to flip a coin that I know is completely fair. I would say that the probability of heads if 50%.

b) You have flipped a coin that I know is completely fair. Nobody knows what has been the result. I would say that the probability of heads is 50%.

c) You have flipped a coin that I know is completely fair. You know the result but I don't. I would say that the probability of heads is 50%.

In some cases you would say that I'm 100% right on my assesment of the probability being 50% while in others the actual probability is either 100% (with probability 50%) or 0% (with probability 50%). This seems irrelevant as far as my statement about the probability being 50% is concerned.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#126

Post by [deleted] »

thaumasiotes wrote: post #121
dasil003 wrote: post #119

Earlier quoted context omitted.

Sure those are just basic stats and could be improved probably, but they do reflect the reality that Brazil should have won; they got unlucky with an own goal, and they made some key mistakes at critical times, failing to finish great chances. You're not going to find a statistical approach that will account for the subtleties that led to this outcome. The problem with soccer stats in general is that everything hinge…

> You're not going to find a statistical approach that will account for the subtleties that led to this outcome. The problem with soccer stats in general is that everything hinges on low-frequency events based on subtle differences of timing and space. I think this deserves to be elaborated a bit: a game in which 1 is a good score, and often a game-winning score, is never going to be accurately predicted based on a s…

[deleted]

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#127

Post by thaumasiotes »

ghayes wrote: post #124
jtolmar wrote: post #120

Earlier quoted context omitted.

Say I have two models - model A returns around 20% likelihood that the top team wins the world cup, and model B returns around 80% likelihood. I use both of the modeling techniques a few thousand times in various parallel universes, and both of them are exactly right - 20% of 20% predictions result in a win, and so on. Despite them both quantifying uncertainty accurately, isn't model B still better?

Think about the actual uncertainty since anything can happen in the game itself. The easiest way to look at this is to play the game 100 times in a row (preferably in parallel universes, as you say). If team A wins in 60% of games, then that caps the ability to predict the result. You can predict a die roll to be 6 with an certainty of 17%. You can’t do any better.

You can do better if you have foreknowledge or retroactive foreknowledge of the outcome of the die roll, which is the obvious suggestion of jtolmar's comment. If I know the recorded outcomes of a sequence of die rolls, I can have models that predict those outcomes to any accuracy I want. But they're not doing it by measuring the uncertainty involved in prospectively rolling the die.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#128

Post by IkmoIkmo »

You'd have to run this world cup thousands of times by simulation, running it a single time and determining the results are not in line with the model is meaningless and silly.

It's as silly as saying my claim for the odds of nearly perfectly modelling a coin toss (approximately 50/50%) is wrong because a series of 10 coin tosses show different results from my model. The model is not any less correct.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#130

Post by jtolmar »

ghayes wrote: post #124
jtolmar wrote: post #120

Earlier quoted context omitted.

Say I have two models - model A returns around 20% likelihood that the top team wins the world cup, and model B returns around 80% likelihood. I use both of the modeling techniques a few thousand times in various parallel universes, and both of them are exactly right - 20% of 20% predictions result in a win, and so on. Despite them both quantifying uncertainty accurately, isn't model B still better?

Think about the actual uncertainty since anything can happen in the game itself. The easiest way to look at this is to play the game 100 times in a row (preferably in parallel universes, as you say). If team A wins in 60% of games, then that caps the ability to predict the result. You can predict a die roll to be 6 with an certainty of 17%. You can’t do any better.

Say I have a bag of dice one of each of the usual D&D denominations (d4, d6, d8, d10, d12, d20). I draw one at random, ask the models for predictions, and roll it. Model A ignores the information about which one I drew, and predicts a correct distribution of rolls (12.9% chance of rolling a 6). Model B correctly processes the information about which one I drew, and predicts a correct distribution given that information (I drew the d6 so 17% chance of rolling a 6). Both models give correct results overall, but Model B has higher probabilities on average, and I would say it is a better model.

A model should be judged both on how accurately it characterizes its uncertainty and how much evidence it's able to successfully make use of.

Post reply on HN