Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The well-calibrated Bayesian [pdf] (1982)

fitelson.org

Post reply on HN
Print view View original

31–35 of 35 posts

Re: The well-calibrated Bayesian [pdf] (1982)

#31

Post by PaulHoule »

edbaskerville wrote: post #27
wosk wrote: post #11

Earlier quoted context omitted.

The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution. So everything in the paper is distribution and when you forecast for a binary ev…

Emphasizing this response. Bayesian models can always produce simple probabilities if you ask them to. E.g., given this data, what is the probability that the next flip is heads? The fact that the model is represented as a distribution over Bernoulli parameter p doesn't contradict this: you just integrate over the posterior.

I got dragged kicking and screaming into the Bayesian world in the early 2010s when I was trying to bring back the old AI and came to the conclusion that the real problem with the old AI was lack of a systematic approach to reasoning about uncertainty and not the cost of knowledge base maintenance (e.g. that could be brought down by orders of magnitude, the value created by knowledge bases could be brought up)

One problem is aggregating information over multiple steps in the reasoning chain the other problem is the powerset problem. The point probability estimate from integrating over the posterior wasn't useful for the first, so I didn't want it. The second problem is impossible in theory but possible in practice, as the existence of intelligence proves.

Re: The well-calibrated Bayesian [pdf] (1982)

#32

Post by mitthrowaway2 »

mitthrowaway2 wrote: post #30
wosk wrote: post #26

Earlier quoted context omitted.

yes and in my example the ending point of the calculation is 0.58. I make an intermediate step explicit in this derivation because it is important for the understanding of the problem. This kind of intermediate step happens a lot in forecasting competitions, where participants are asked for their forecasts and their confidence in their forecasts. I want to show here that you need to include that in your forecast (all…

But again this seems like a TypeError. If you're asking me for my forecast and my confidence in my forecast, I would say this: Forecast: It will rain tomorrow. Confidence: 0.7 If you say "No no, I mean what is your confidence in the 0.7 number" then I have no idea what you're talking about. 0.7 is my confidence. It's not valid to attach a confidence to that number. When the next day comes, we simply find out whether…

Addendum: The closest thing I can think of that matches the GP's description is conditional probabilities.

For example, if it rains tonight then I am 70% sure it will rain tomorrow too. I am also 80% sure it will rain tonight. And I am 100% sure that if it doesn't rain tonight, it also will not rain tomorrow. Then I can chain these together: The chance of rain tomorrow is P(Rain tomorrow) = P(Rain tomorrow | rain tonight) * P(Rain tonight) = 0.8*0.7 = 0.56.

You can assign conditional probabilites to the correctness of your model itself. Then the phrasing is more like "I am 70% confident that it will rain tomorrow, conditional on my understanding of meteorology being correct" and "I am 80% sure that I understand meteorology correctly". Then you also need to add in a term for P(Rain | not understanding correctly).

Re: The well-calibrated Bayesian [pdf] (1982)

#33

Post by oliculipolicula »

bryanrasmussen wrote: post #13
PaulHoule wrote: post #10

Earlier quoted context omitted.

If he was well calibrated there is no way they would have made it through 79 episodes!

Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.

To his defence. in the trekkie timeline, neither Vulcans nor humans had discovered the Metropolis algorithm, so Spock had (for sure) nothing with which to calibrate that

Re: The well-calibrated Bayesian [pdf] (1982)

#34

Post by wosk »

mitthrowaway2 wrote: post #32
mitthrowaway2 wrote: post #30

Earlier quoted context omitted.

But again this seems like a TypeError. If you're asking me for my forecast and my confidence in my forecast, I would say this: Forecast: It will rain tomorrow. Confidence: 0.7 If you say "No no, I mean what is your confidence in the 0.7 number" then I have no idea what you're talking about. 0.7 is my confidence. It's not valid to attach a confidence to that number. When the next day comes, we simply find out whether…

Addendum: The closest thing I can think of that matches the GP's description is conditional probabilities. For example, if it rains tonight then I am 70% sure it will rain tomorrow too. I am also 80% sure it will rain tonight. And I am 100% sure that if it doesn't rain tonight, it also will not rain tomorrow. Then I can chain these together: The chance of rain tomorrow is P(Rain tomorrow) = P(Rain tomorrow | rain ton…

yes exactly, it's a conditional forecast. I intended: P(Rain tomorrow | my model is right) = 0.7 p(my model is right) = .8 p(my model is wrong) = .2 p(rain tomorrow | my model is wrong) = something_you_believe_about (= 0.1 in my example) 0.80.7 + 0.2 0.1 = 0.58

you may replace "my model is wrong" by e.g " my certainty that no one is using cloud-activation technology".

Now, I'm not native in English and I see how someone (especially with a math background) might find this sentence confusing:

> you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast.

Turns out it's very hard about math non-formally for generalist audience and might not have done the best job about it. It should read

> you are predicting something (e.g rain tomorrow), then your model believe it will rain with probability 0.7 but you are 80% sure of your model, then you won’t report 0.7; you will report : 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast.

Post reply on HN