Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goldman Sachs model to predict World Cup game results didn’t come close

bloomberg.com

Post reply on HN
Print view View original

101–110 of 136 posts

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#101

Post by lordnacho »

TuringNYC wrote: post #94
soVeryTired wrote: post #84

Earlier quoted context omitted.

Surely your book is meant to be hedged though?

Well, you've got exposures on both sides (and you keep the tiny bit in difference between what you buy at and what you sell at, in aggregate.) I'm not sure how gambling bookies work, but I'm assuming it is very simple principles...arent gambling bookies essentially market makers just like Wall Street market makers?

Yeah, when I started as a trader it was pretty much a given that you'd be interested in poker and sports betting. The "training" would consist of the the senior traders asking you to make markets in just about every imaginable sports event, an after-work poker game, and a good few questions about things that aren't normal bets (how many burgers do you think you could eat?).

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#102

Post by jbob2000 »

cepth wrote: post #61
jbob2000 wrote: post #49

Earlier quoted context omitted.

But Goldman Sachs are the kings of predicting uncertainty! This is their whole business! They make billions predicting certainty through the murky, uncertain waters of the global economy. Would you argue that the global economy is more uncertain that soccer? I'd say so. How is it that they can find success in the market but not in soccer? I think this is a smoke signal. Soccer is corrupt; you can't predict the winner…

Unclear if your comment is tongue in cheek, but assuming that you're serious, I'd encourage you to give a listen to a podcast episode like this: https://soundcloud.com/bettheprocess/episode-35-ted-knutson . In the world of sports betting/analytics, you have baseball and basketball at the forefront, and then American football, soccer, and hockey (roughly in that order). Off the top of my head, there are several reason…

Ok, I understand this - that soccer has many variables and it is difficult to create a model with all of these variables. But my point is, the global economy has way more variables than soccer. Way way way way more variables. At least 7.5 billion of them.

So would you argue that creating a statistical model of soccer is harder than creating one for global economies? I think it's harder to model economies.

I'm not even trying to give Goldman a hard time! I'm saying that Goldman probably put together a very accurate model of "soccer", but we aren't watching an accurate model of soccer; we're watching the corrupted one where the players and skills don't matter.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#103

Post by cepth »

jbob2000 wrote: post #102
cepth wrote: post #61

Earlier quoted context omitted.

Unclear if your comment is tongue in cheek, but assuming that you're serious, I'd encourage you to give a listen to a podcast episode like this: https://soundcloud.com/bettheprocess/episode-35-ted-knutson . In the world of sports betting/analytics, you have baseball and basketball at the forefront, and then American football, soccer, and hockey (roughly in that order). Off the top of my head, there are several reason…

Ok, I understand this - that soccer has many variables and it is difficult to create a model with all of these variables. But my point is, the global economy has way more variables than soccer. Way way way way more variables. At least 7.5 billion of them. So would you argue that creating a statistical model of soccer is harder than creating one for global economies? I think it's harder to model economies. I'm not eve…

I think we have to be very clear on what economic "models" Goldman uses.

If you're talking about GDP growth forecasting, or forecasting unemployment numbers, these are ultimately questions of aggregation. Yes, there are 7.5 billion people, but at the end of the day each individual agent's actions don't make a tremendous difference for an aggregate measure like GDP. During periods of low volatility, as we are currently experiencing, it's really not all that impressive to forecast the unemployment rate +/- 0.25%, or GDP growth within 0.5%.

If you're taking about their market-making and trading businesses, they've had some horrendous quarters recently as well (http://www.businessinsider.com/goldman-sachs-just-had-a-hist...). A very small portion of Goldman's business is taking an opinionated stance, most of their income comes through relatively low-risk market making activities.

And let's not forget that during the 2008 financial crisis, certain departments within the company correctly wagered against credit default swaps, while others had exposure to subprime mortgages. The company still needed an injection of capital from Warren Buffett and the US Treasury to weather the crisis. Point being, they aren't clairvoyant oracles.

---

Regarding your last point, which was also made in your original comment, you seem to be claiming some form of what economists call "omitted variable bias", and seem to be hypothesizing that the "omitted variable" is corruption or cheating.

From the purely technical standpoint of building models, the tiny samples (https://www.theringer.com/soccer/2018/7/11/17557720/world-cu...) and the nature of the "data" being collected means that there are plenty of other explanations, like incorrectly estimated parameters or measurement error.

If you're trying to suggest that there is corruption or cheating in soccer, please point to a concrete example of a team in a critical game receiving a disproportionate number of calls. Unsure if you're aware, but this was the first World Cup with instant video replays for the referees to use. Had this replay been in use more widely in international soccer, the US might've qualified for this World Cup (https://deadspin.com/u-s-a-out-of-world-cup-on-phantom-goal-...), England might've won/tied that pivotal 2010 World Cup game (https://en.wikipedia.org/wiki/Ghost_goal#England_v_Germany_a...), etc.

Soccer may have had a sordid past with the picking of host countries, but the trends in the actual game itself point to technology reducing the ability of referees to make blatantly terrible calls.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#104

Post by thousandautumns »

bambax wrote: post #97
yk wrote: post #30

> And in any case, the model only generated probabilities of winning a game and advancing, and no team was given more than an 18.5 percent chance of winning the World Cup. > [...] > But Goldman Sach’s misfire is perhaps the most curious. The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win,…

> The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win, and that that is the highest chance does not say much about the model. But do you need a sophisticated model and lots of so-called "AI" to arrive at the conclusion that there's a lot of uncertainty?? The point of the model is to reduce…

The point of the model is absolutely not to reduce uncertainty, it is to quantify it, which are two very different things. No model reduces uncertainty in a probabilistic sense.

And no, you don’t need statistics or machine learning to say “there is a lot of uncertainty”, but you do in order to quantify that uncertainty.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#105

Post by LeifCarrotson »

CamTin wrote: post #89
LeifCarrotson wrote: post #75

Earlier quoted context omitted.

> Predicting the result of an A or B contest the bar is already defined. I disagree: If team A has a 10-30% chance of winning, and A pulls off the upset, the correct answer was not "A Wins" it was "B has a 70-90% chance of winning". For Goldman Sachs' investments, the bar is not to predict that A wins or that B wins, it's to predict the probability and variance regarding which team will win. Of course, from a single…

The problem is that the 2018 World Cup is not a repeatable event. Neither are most open-market trades (presumably the point of this whole PR stunt being to show that their quants are good at making smart bets in the markets) but they're a LOT closer. Soccer is a pretty data-poor environment, or at least was historically. Before movement trackers, there was very little data to play with. With movement tracking data sl…

> The problem is that the 2018 World Cup is not a repeatable event. Neither are most open-market trades...

The 2018 World Cup is not a repeatable event, Elon Musk buying $10M of Tesla shares is not a repeatable event, and Donald Trump winning the 2016 presidential election is not a repeatable event. Therefore, to meaningfully discuss any of these in the context of probabilities and confidence intervals, we must assume that we generalize them to any soccer game, a stock purchase, or an election, and can do this meaningfully by adjusting our priors. It does make the mathematics a lot less pure.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#106

Post by thousandautumns »

CamTin wrote: post #89
LeifCarrotson wrote: post #75

Earlier quoted context omitted.

> Predicting the result of an A or B contest the bar is already defined. I disagree: If team A has a 10-30% chance of winning, and A pulls off the upset, the correct answer was not "A Wins" it was "B has a 70-90% chance of winning". For Goldman Sachs' investments, the bar is not to predict that A wins or that B wins, it's to predict the probability and variance regarding which team will win. Of course, from a single…

The problem is that the 2018 World Cup is not a repeatable event. Neither are most open-market trades (presumably the point of this whole PR stunt being to show that their quants are good at making smart bets in the markets) but they're a LOT closer. Soccer is a pretty data-poor environment, or at least was historically. Before movement trackers, there was very little data to play with. With movement tracking data sl…

Literally no data producing phenomenon is a repeatable event, outside of controlled experiments.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#107

Post by jbob2000 »

cepth wrote: post #103
jbob2000 wrote: post #102

Earlier quoted context omitted.

Ok, I understand this - that soccer has many variables and it is difficult to create a model with all of these variables. But my point is, the global economy has way more variables than soccer. Way way way way more variables. At least 7.5 billion of them. So would you argue that creating a statistical model of soccer is harder than creating one for global economies? I think it's harder to model economies. I'm not eve…

I think we have to be very clear on what economic "models" Goldman uses. If you're talking about GDP growth forecasting, or forecasting unemployment numbers, these are ultimately questions of aggregation. Yes, there are 7.5 billion people, but at the end of the day each individual agent's actions don't make a tremendous difference for an aggregate measure like GDP. During periods of low volatility, as we are currentl…

Thanks for the replies and the detailed sources, it's interesting to read!

> Point being, they aren't clairvoyant oracles.

Yeah, my argument was weak in that regard. They aren't anywhere close to perfect or accurate, I'll admit.

> you seem to be claiming some form of what economists call "omitted variable bias"

Yes! Is that what it's called?

> please point to a concrete example of a team in a critical game receiving a disproportionate number of calls

Corruption doesn't have to be that explicit. Maybe key players or coaches are paid to perform poorly? It doesn't always come down to the ref. But I admit I have no examples.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#108

Post by afterburner »

corpMaverick wrote: post #86
raverbashing wrote: post #21

People conflate statistics with actual results more often than not and I think those reporting on such stories and maybe even the original authors might fall for this. It was not wrong to say Hillary had a 95% chance of winning the presidential election, but the confidence was low and that value still allowed for the opposite result to happen . Also football has a lot of variance concerning team capability and end re…

According to 538 Hillary had a 70% chance. Yes. Many people completely interpreted that like 70% would vote for Clinton. I had to explain the meaning of "70% chance" to people who should know better. I think they just heard the number and didn't give it a second thought.

In general, people don't think.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#109

Post by sandov »

thousandautumns wrote: post #104
bambax wrote: post #97

Earlier quoted context omitted.

> The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win, and that that is the highest chance does not say much about the model. But do you need a sophisticated model and lots of so-called "AI" to arrive at the conclusion that there's a lot of uncertainty?? The point of the model is to reduce…

The point of the model is absolutely not to reduce uncertainty, it is to quantify it, which are two very different things. No model reduces uncertainty in a probabilistic sense. And no, you don’t need statistics or machine learning to say “there is a lot of uncertainty”, but you do in order to quantify that uncertainty.

I think the right way to measure the correctness of the model is to compare it with various other predictions:

-Predictions from the general public

-Predictions from football experts

-Predictions from other mathematical models

For example: If over time, the new model is 5% better than the best of the old models, then it's very good.

Doesn't make much sense to compare it with reality and jump to the conclussion that the model doesn't work because no prediction can be 100% accurate.

Re: Goldman Sachs model to predict World Cup game results didn’t come close

#110

Post by satsuma »

CamTin wrote: post #89
LeifCarrotson wrote: post #75

Earlier quoted context omitted.

> Predicting the result of an A or B contest the bar is already defined. I disagree: If team A has a 10-30% chance of winning, and A pulls off the upset, the correct answer was not "A Wins" it was "B has a 70-90% chance of winning". For Goldman Sachs' investments, the bar is not to predict that A wins or that B wins, it's to predict the probability and variance regarding which team will win. Of course, from a single…

The problem is that the 2018 World Cup is not a repeatable event. Neither are most open-market trades (presumably the point of this whole PR stunt being to show that their quants are good at making smart bets in the markets) but they're a LOT closer. Soccer is a pretty data-poor environment, or at least was historically. Before movement trackers, there was very little data to play with. With movement tracking data sl…

wasn't leicester city a "moneyball" team? a zero-to-hero club with a roster of modest salaried players who have statistical synergy? i don't follow much premier league but from what i remember hearing about it, they bucked a trend of spending tens/hundreds of millions for megastars to solo carry the team
Post reply on HN