Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Better Crystal Ball?

foreignaffairs.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: A Better Crystal Ball?

#31

Post by vosper »

jacques_chester wrote: post #12
vosper wrote: post #8

> Well-calibrated forecasters, for instance, can estimate the likelihood that a skirmish with the Chinese navy in the South China Sea will result in at least two American deaths by December 31. But what policymakers really want to know is the extent to which China will threaten U.S. interests in the coming years and decades. This has never happened (a skirmish, let alone deaths) so how can we think that the forecaste…

> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat act…

Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right?

The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen...

Perhaps I misinterpreted the paragraph?

Re: A Better Crystal Ball?

#32

Post by cl42 »

vosper wrote: post #31
jacques_chester wrote: post #12

Earlier quoted context omitted.

> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat act…

Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel l…

In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y".

In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not.

There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult.

At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.

Re: A Better Crystal Ball?

#33

Post by cl42 »

vosper wrote: post #29
jessriedel wrote: post #21

Earlier quoted context omitted.

There's strong empirical evidence that prediction ability cross generalizes. That is, if a person is good at predicting unrelated topics A, B, and C, they are also like to be good at topic D.

But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here) The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?

Also worth noting: it's about being better than an existing baseline. Effectiveness is measured in relation to a baseline accuracy, which can already be quite poor.

Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)

Re: A Better Crystal Ball?

#34

Post by vosper »

cl42 wrote: post #32
vosper wrote: post #31

Earlier quoted context omitted.

Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right? The article seems to be suggesting that this can be accurately estimated, but I feel l…

In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to…

Thanks for the explanation, this helped me understand the idea a bit better.

Re: A Better Crystal Ball?

#35

Post by tim333 »

trishankkarthik wrote: post #3

This is the most dangerously stupid thing I've ever heard. Tetlock is a known academic charlatan pushing his absolutely useless "superforecasting" nonsense which @nntaleb keeps debunking on Twitter. If Tetlock and friends are so good about forecasting the future, why didn't they predict and warn us about COVID-19 BEFORE @nntaleb and friends (including myself) did? P.S. I have been downvoted for saying this.

Thanks for the context. I was trying to figure what was noteworthy about an article saying we should think ahead which I kind of assumed we did anyway. Though I think "most dangerously stupid thing I've ever heard" is maybe an exaggeration. The US govt effectively banning coronavirus testing back in Feb kind of sticks out in my mind as outstanding on that front (https://www.nytimes.com/2020/03/10/us/coronavirus-testing-de...)

Re: A Better Crystal Ball?

#36

Post by trishankkarthik »

tim333 wrote: post #35
trishankkarthik wrote: post #3

This is the most dangerously stupid thing I've ever heard. Tetlock is a known academic charlatan pushing his absolutely useless "superforecasting" nonsense which @nntaleb keeps debunking on Twitter. If Tetlock and friends are so good about forecasting the future, why didn't they predict and warn us about COVID-19 BEFORE @nntaleb and friends (including myself) did? P.S. I have been downvoted for saying this.

Thanks for the context. I was trying to figure what was noteworthy about an article saying we should think ahead which I kind of assumed we did anyway. Though I think "most dangerously stupid thing I've ever heard" is maybe an exaggeration. The US govt effectively banning coronavirus testing back in Feb kind of sticks out in my mind as outstanding on that front ( https://www.nytimes.com/2020/03/10/us/coronavirus-test…

Thanks, yes, I try to write in a way where even I don't fall asleep, unlike our "nirvana fallacy" friend below...

Re: A Better Crystal Ball?

#37

Post by jessriedel »

vosper wrote: post #29
jessriedel wrote: post #21

Earlier quoted context omitted.

There's strong empirical evidence that prediction ability cross generalizes. That is, if a person is good at predicting unrelated topics A, B, and C, they are also like to be good at topic D.

But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once? Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here) The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?

> But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once?

No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs.

The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow.

You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless.

The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.

Re: A Better Crystal Ball?

#38

Post by jacques_chester »

cl42 wrote: post #2

I’ve been doing something similar for investing in stocks and it has worked well. I’m exploring building an NLP-powered version of this (based on crawling news events). If anyone is interested in chatting or collaborating, let me know. :)

Anecdote time: an NLP specialist I once worked with told me about some research he did into determining the "goodness" or "badness" of press releases and SEC filings. Buy on good news, sell on bad news, hopefully fast enough to beat others to it.

After a great deal of crunching and study he said they did come up with a model that could do it reasonably well. Then realised it could be replaced with a simple rule:

Are there a bunch of numbers at the top? Good news.

Are the numbers buried way down? Bad news.

Another simple rule, by the way, is timing vs executive compensation events. If there's an announcement just before a big block of options or RSUs are vesting, then it's good likely to be good news. If it's being posted far away from vesting dates, then it's likely to be bad news.

Re: A Better Crystal Ball?

#39

Post by jacques_chester »

trishankkarthik wrote: post #27
jacques_chester wrote: post #11

Earlier quoted context omitted.

Taleb has anecdotes and insults. Tetlock has data piled on data.

IDK how many more times I have to repeat this before it gets through your head that one warned about the pandemic blowing up before it happened, therefore only one can be taken seriously. You don't need evidence to run away when someone shouts that a bear is coming your way.

"There will be a pandemic" is a prediction with 100% accuracy. Any rare event, as Taleb likes to hang his entire career on, will eventually happen.

But such a prediction is basically worthless without a horizon. Information about when a pandemic happens affects what actions have to be made to deal with it. Pandemics that definitely happen tomorrow require far more expensive and disruptive actions than ones happening sometime in the next decade with gradually increasing probability.

Tetlock's work is about people who actually have to define a date and for whom the deciding measurements are objective. You can't just say "a pandemic!", you have to say "a pandemic is declared by WHO on or before January 1st, 2019".

In fact, Tetlock's data undermines Taleb's entire thesis that dramatic events are systematically under-predicted. He showed that dramatic scenarios are over-predicted by experts. They have higher emotional salience, and history that is taught focuses on dramatic, outlier events because of their disproportionate impact. Due to availability bias and hindsight bias, experts typically predict that dramatic events will happen more frequently than they actually do. Such predictions are regularly roflstomped by hilariously simple forecasting methods like "yesterday's weather" or fitting a line on a handful of data points.

As I said earlier: Taleb has anecdotes and insults. Tetlock has data.

Re: A Better Crystal Ball?

#40

Post by jacques_chester »

vosper wrote: post #34
cl42 wrote: post #32

Earlier quoted context omitted.

In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y". In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not. There is some art to this -- the "event" in question needs to…

Thanks for the explanation, this helped me understand the idea a bit better.

Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right.

So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best).

There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_pag...

Post reply on HN