71–80 of 136 posts
If anything, it worked worse.
"If anything"? All the results are available, so it would be easy to put a precise number on this. Measure the Bayesian regret, or just report the winnings if you had used the GS model to bet on the outcomes. Unless it reports some concrete numbers, this article is garbage.
It doesn't report any concrete numbers.
Groups Round_16 Quarters Semis Finals Out_In Probability
Brazil 87.5% 60.8% 42.0% 27.9% 18.5% Quarters 18.8%
France 81.4% 58.4% 36.6% 19.9% 11.3% Won 11.3%
Germany 80.5% 49.5% 30.5% 18.8% 10.7% Groups 19.5%
Portugal 75.2% 52.8% 32.2% 17.3% 9.4% Round_16 22.4%
Belgium 78.5% 51.1% 27.7% 15.8% 8.2% Semis 11.9%
Spain 72.3% 50.1% 28.8% 15.4% 7.8% Round_16 22.2%
England 73.1% 46.6% 24.4% 13.4% 6.5% Semis 11.0%
Argentina 79.7% 44.2% 24.1% 11.8% 5.7% Round_16 35.5%
Colombia 74.9% 37.3% 17.0% 8.5% 3.7% Round_16 37.6%
Uruguay 74.4% 34.6% 17.2% 7.2% 3.2% Quarters 17.4%
Poland 68.5% 30.5% 12.8% 5.8% 2.3% Groups 31.5%
Denmark 47.8% 26.3% 12.4% 5.2% 2.0% Round_16 21.5%
Mexico 52.0% 23.2% 10.5% 4.9% 1.9% Round_16 28.8%
Sweden 45.9% 19.4% 8.3% 3.7% 1.3% Quarters 11.1%
Iran 35.4% 18.1% 7.2% 2.6% 0.8% Groups 64.6%
Peru 37.3% 17.2% 6.8% 2.5% 0.8% Groups 62.7%
Australia 33.5% 15.4% 6.3% 2.3% 0.7% Groups 66.5%
Russia 47.9% 16.3% 6.0% 2.0% 0.7% Quarters 10.3%
Croatia 49.8% 16.9% 6.3% 2.1% 0.6% Finals 4.2%
Switzerland 52.8% 15.9% 6.1% 2.0% 0.6% Round_16 36.9%
Iceland 45.2% 15.1% 5.6% 1.8% 0.5% Groups 54.8%
Costa_Rica 36.8% 13.3% 4.7% 1.6% 0.5% Groups 63.2%
Serbia 32.9% 12.1% 4.5% 1.5% 0.5% Groups 67.1%
Japan 36.5% 12.8% 3.8% 1.3% 0.4% Round_16 23.7%
Saudi_Arabia 43.4% 12.7% 4.2% 1.3% 0.4% Groups 56.6%
Tunisia 35.2% 13.3% 4.1% 1.3% 0.4% Groups 64.8%
Egypt 34.4% 8.7% 2.5% 0.7% 0.2% Groups 65.6%
South_Korea 21.6% 5.9% 7.1% 0.5% 0.2% Groups 78.4%
Morocco 17.1% 6.8% 1.8% 0.5% 0.1% Groups 82.9%
Nigeria 25.2% 6.5% 1.7% 0.4% 0.0% Groups 74.8%
Senegal 20.1% 4.9% 1.2% 0.3% 0.0% Groups 79.9%
Panama 13.2% 3.3% 0.5% 0.1% 0.0% Groups 86.8%
Edit: Fix copy-paste errors and atrocious maths.
I watched quite a few matches and among the things I saw in the matches but not in any statistics are: - motivation (Germany and Croatia were the two extremes here, no idea how to measure it) - team cohesion (number of articles in a few journals questioning the team cohesion, maybe also articles about individual players) - creativity in offense (maybe measurable via "target missed from close distance" + "ball passed…
These kinda show what makes predicting football particularly difficult. I like the ideas, and I think we (or more likely some ML algorithm) can come up with the set of conditions that showed why France prevailed against the specific opposition at this specific World Cup ... but I suspect that the conditions would be pretty unique and invalid for Euro 2020, WC 2022 etc. As you identified, motivation could be pretty ha…
> I'm not sure what you mean with the last one, but I think this could be a nice one - if you mean "times you lost possession in your own half"
Almost, England lost the ball frequently (> 50+x% with a large x AFAI could see) due to the keeper sending out long balls. I'd like to measure that somehow. Could be done via number of seconds in possession after a goal kick, an indicator whether a hypothetical 85% marker of the field was reached or measuring whether the ball was at least 5x successfully passed (or resulted in a goal).
> And in any case, the model only generated probabilities of winning a game and advancing, and no team was given more than an 18.5 percent chance of winning the World Cup. > [...] > But Goldman Sach’s misfire is perhaps the most curious. The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win,…
Uncertainty is a truism; that's why people want to use a prediction algo. Did the system so better on results it was more certain about? Predicting the result of an A or B contest the bar is already defined. Either the system gets it right or doesn't, if it gets it right more often than not then (despite this being poor grounds mathematically, on a small result pool) popular press will report it as successful. IMO if…
I disagree: If team A has a 10-30% chance of winning, and A pulls off the upset, the correct answer was not "A Wins" it was "B has a 70-90% chance of winning".
For Goldman Sachs' investments, the bar is not to predict that A wins or that B wins, it's to predict the probability and variance regarding which team will win. Of course, from a single upset game, it's impossible to tell whether these estimates are correct. You'd need to see the success or failure of many trials.
> And in any case, the model only generated probabilities of winning a game and advancing, and no team was given more than an 18.5 percent chance of winning the World Cup. > [...] > But Goldman Sach’s misfire is perhaps the most curious. The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win,…
Their model also had France at 2nd most likely, Belgium at 5th, and England at 7th. 3 of their top 7 made the Semi-Finals, and they called the eventual winner as Second Most Likely, and more likely than Germany. They actually predicted the Brazil/Belgium game in the Quarter Finals, but got the winner wrong. Brazil had 27 shots and 9 on target with 59% posession. Belgium only had three shots on target, and made two of…
A modern model would accommodate for the fact that those numbers alone mean nothing, because they don't. Those are the numbers broadcasters reluctantly put on a screen for entertainment value, but they don't have real analytical power because they have no comparative metric.
How up or down were each of those numbers against previous wins and losses for each team?
What was Brazil's conversion from on-target shots before the tournament?
What was Belgium's success/failure rate on on-target shots they were defending against?
Likewise the other way around: were Brazil guilty of particularly poor defending? Were Belgium finding ways of making on-target shots count against all opposition, or was it luck on this game?
Any human analyst could tell you going into that game that Belgium were "lucky" and easily free scoring beyond expectations, able to make more of fewer opportunities. Likewise the consensus from most experts was that Brazil were guilty of mild complacency, the team were young and not yet formed into a strong unit yet (rather still just 11 strong individuals at any one point in time), and their on-target shots - whilst frequent - were of lower probability of being able to turn into goals due to distance, power, position, etc.
So why did the Bloomberg model not pick that up?
I actually think they did pretty well all things considering, but I'd love to see whether they did any runs on previous World cups to try and check their thinking and whether they over-fitted a little to a couple of key metrics. I think the lack of metrics from previous games might mean they relied on some headline numbers, but there's more that they could have done to get a better model here...
Still, it's not their job is it? Just a bit of fun... which is a good job, because I find it just a little bit amusing.
Duh. Looks like there's a fundamental misunderstanding of how statistics works all around. The probability of an event does NOT predict a particular outcome. Ever. It only says that if the experiment is performed again and again and again, like a few thousand times, then X% of those will match that probability. If I toss a fair coin you cannot predict the next outcome. You can only say that if I toss the coin a 1000…
If you read the actual report they did[0], they never claimed that any single outcome was more than 18.5% likely.
[0]: http://www.goldmansachs.com/our-thinking/pages/world-cup-201...
I compared the logloss for their predictions with the "uniform" benchmark (giving each team 1/32 probability of winning, 1/16 probability of getting to the finals, etc) and the results are the following (if I transcribed the data properly):
Getting to second round:
GS: 0.495 UBS: 0.495 bench: 0.693
Getting to quarter-finals:
GS: 0.463 UBS: 0.459 bench: 0.562
Getting to semi-finals:
GS: 0.310 UBS: 0.327 bench: 0.377
Getting to final:
GS: 0.231 UBS: 0.269 bench: 0.234
World-cap winner:
GS: 0.097 UBS: 0.113 bench: 0.139
The performance of the models was ok until Croatia got to the finals. This hurt specially UBS, who predicted less than 0.9% probability of such an event (compared to 2.1% in Goldman's model).
Edit: these would have been the "best case" scores (if the high-probabilty teams had classified to each round, ignoring that this may be impossible due to the structure of the tournament):
GS: 0.432 0.302 0.220 0.141 0.079
UBS: 0.365 0.251 0.176 0.111 0.070
UBS could potentially achive lower logloss metrics because it had more extreme predictions.
Earlier quoted context omitted.
FYI - I worked at Goldman Sachs and then a hedge fund for a decade. On the Capital Markets / Trading side, you are literally a bookie. In fact the nomenclature is "you have a book." You are setting trading spreads based on where you think things will go. Depending on the market, your work may be more or less statistical and you're trying to gain a statistical advantage.
Off-topic: were you able to retire after that decade?
I think people have a strange view of finance. Most people aren't paid obscene amounts of money in finance, just like most software developers don't make the salary of a senior developer at Big Tech. They also work an obscene amount of hours. During earnings season, I would be at my desk by 5am and work 80+ hours per week. Nowadays, It's a rarity to go more than 50. My brother currently works at Big Bank, and makes more than I do on an absolute basis, but I definitely make more than he does hourly. I get to work at 9:30-10, he gets to work at 7:30-8. I get home at 6:30-7:00, he gets home 8:30-9. He works at least a half day every Sunday, I enjoy my hobbies. I'm also commenting on HN at 11:00...
Most of my college friends still work in finance. I make more money than a few of them based on overly honest drunken conversations, and we're all more than 10 years into our careers. There is a glass ceiling in tech that is a lot more all-encompassing, but it's not like it doesn't exist in other industries. There are only so many higher-up positions, and most people burn out (or aren't capable of competing) before they even get in position for that promotion. The running joke when someone was getting poor performance reviews was "That's it, I'm moving to Vermont to open an antique store".
For some more comparison, I grew up in a 1%er town in the suburbs of NY. The average lawyer family lived in nicer houses than the average finance family, who in turn lived in nicer houses than the average medicine family. However, the most expensive house was owned by the CFO of Big Bank. Income is very right-skewed in finance.
Earlier quoted context omitted.
FYI - I worked at Goldman Sachs and then a hedge fund for a decade. On the Capital Markets / Trading side, you are literally a bookie. In fact the nomenclature is "you have a book." You are setting trading spreads based on where you think things will go. Depending on the market, your work may be more or less statistical and you're trying to gain a statistical advantage.
Off-topic: were you able to retire after that decade?
Long Answer: Full retirement is hard, Healthcare is a pain in the US. You cant really "save" for it in the US, it can swallow all your savings, so you'll always need some job or another to cover healthcare and catastrophic needs. That said, you can very easily down-shift once you have a house, savings, etc.
Longer Answer: Could have, if I wanted to -- but you always give something up in exchange. These jobs will take everything you give them (time, health, life) and give back a decent percentage (income.) But you cannot easily dial up or down the work, it comes in chunks and you have to complete it. My life was increasingly unhinged at 27 and I decided to jump off the treadmill after seeing a colleague continue to work through his mother's terminal illness and death. Inertia and greed are a toxic combination. Numerous colleagues were on drugs, uppers, anti-depressants, etc. One died from stress (heart attack in his 30s.)
I chose to get married, have two kids. I switched to a pure tech job (now an ML product owner at a Series A pure tech firm.) We have dinner together almost every single day. Weekends are completely ours. We go to the park most warm days. We take 3 to 4 vacations a year, many with my mom as well. There is a decent amount of work but I can choose when to do it (unlike Wall St.) and the work is longer term and I can dial it up/down as family requires. I sit outside and read during lunch. I turn off the markets when i step out of work.
Many of my colleagues were easy millionaires by ~30 and multi-mullionaires if they stuck till their mid 30s and were focused. Many others blew through their bonuses (or snorted it away) and ended up with nothing and just live bonus to bonus. It also depends on the job (business/deal side vs quant side vs tech -- the money is a waterfall across the 3 sections.) As with all industries, you get ripped off if you dont fight for your share of the pie. Plenty of people avoid conflict and life comfortable lives and nothing more. I also saw several C++ programmer/manager earn double digit millions of dollars over several years, one earned over 100MM USD over the course of his time at the hedge fund (public records, check out AIG-FP https://en.wikipedia.org/wiki/AIG_bonus_payments_controversy)
I think I did well and hopefully dont have to worry about poverty anymore. You either get lucky (early FB employee, hot product at xyz.com) or you have to give up something. I havent seen someone truthfully say they got both money and family and happiness all together.