Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The well-calibrated Bayesian [pdf] (1982)

fitelson.org

Post reply on HN
Print view View original

21–30 of 35 posts

Re: The well-calibrated Bayesian [pdf] (1982)

#21

Post by mitthrowaway2 »

wosk wrote: post #5

It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert). A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday).…

> if you believe it will rain with probability 0.7 but you are 80% sure of your belief

What does this even mean...?

If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist tomorrow.

Bayesians of course accept that their models can have errors, and if they're doing a good job they'll factor all of the most influential ones into the probability calculation itself.

Re: The well-calibrated Bayesian [pdf] (1982)

#22

Post by danbruc »

glial wrote: post #17
danbruc wrote: post #14

Something Bayesian. Despite my best effort I just do not get Bayesian probability, it more or less just does not make sense to me. Can you convince me otherwise? What is your best example of something with a probability that can not be analyzed in terms of frequencies or other proportions? And your Bayesian account of it must make sense, I am 90 % certain that P != NP and that is why I would take bets based on those…

Someone walks out of a magic store holding a coin. They propose a bet. If they flip it 100 times and the proportion of heads is within [0.4, 0.6], you win $100. If it's not, you pay $100. Do you take that bet? Explanation: absent the magic store scenario, a `rational' person would take the bet. Your prior belief is that most coins are roughly unbiased. Given that they walked out of a magic store, you now have additio…

The bet does not really matter, the central question is whether they have a fair coin or if they are trying to me in some way. Even without the magic store, I would be very suspicious of anyone approaching random people with an offer like that.

So I would certainly consider it likely, that they are trying to trick me. But the probability I would assign to this, would still be rooted in some frequency, somewhere under the hood I would try to estimate the possible situations leading to such an offer and in which fraction of them I will be tricked.

If I am doing a good job with that, then repeatedly being in this situation should result in me getting tricked with the probability I cooked up. If I am bad at figuring out the possible states and their probabilities, then I they will not match.

Re: The well-calibrated Bayesian [pdf] (1982)

#23

Post by mitthrowaway2 »

danbruc wrote: post #19
kgwgk wrote: post #16

Earlier quoted context omitted.

What's the probability that the sinking of the USS Maine in 1898 was accidental?

One could look at the likelihood of spontaneous coal fires and their effects, gather evidence about the activities surrounding the event, essentially trying to narrow down the set of possible states that would lead to the incident and then see what proportion of those states cause the sinking by fire, military action, or something else. But then attaching a number to that seems a dauting task.

Attaching a number to it isn't so daunting, if you view that number as a reflection of your own subjective certainty as to the truth of the proposition. There's no other "correct" number to shoot for; in reality it was either an accident or it wasn't; Nature knows which proposition is true but we don't. Our mission is only to find the figure that minimizes the logarithm of our error, if the truth were at some point to be revealed to us. The more evidence you gather and the more work you put in will get you a better minimum, but there's no reason to be afraid of putting even a rough quantitative number on it before doing that legwork; you'd just be doing it from a state of higher uncertainty, and your stated probability would have to be farther from either confident extreme (0% or 100%) accordingly.

Re: The well-calibrated Bayesian [pdf] (1982)

#24

Post by wosk »

mitthrowaway2 wrote: post #21
wosk wrote: post #5

It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert). A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday).…

> if you believe it will rain with probability 0.7 but you are 80% sure of your belief What does this even mean...? If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist…

> What does this even mean...?

You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).

Re: The well-calibrated Bayesian [pdf] (1982)

#25

Post by mitthrowaway2 »

wosk wrote: post #24
mitthrowaway2 wrote: post #21

Earlier quoted context omitted.

> if you believe it will rain with probability 0.7 but you are 80% sure of your belief What does this even mean...? If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist…

> What does this even mean...? You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).

No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.

Re: The well-calibrated Bayesian [pdf] (1982)

#26

Post by wosk »

mitthrowaway2 wrote: post #25
wosk wrote: post #24

Earlier quoted context omitted.

> What does this even mean...? You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).

No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.

yes and in my example the ending point of the calculation is 0.58.

I make an intermediate step explicit in this derivation because it is important for the understanding of the problem.

This kind of intermediate step happens a lot in forecasting competitions, where participants are asked for their forecasts and their confidence in their forecasts. I want to show here that you need to include that in your forecast (all you belief) and not keep it separated.

Re: The well-calibrated Bayesian [pdf] (1982)

#27

Post by edbaskerville »

wosk wrote: post #11
PaulHoule wrote: post #8

Earlier quoted context omitted.

I didn't see a real Bayesian point of view in that article. A Bayesian does not give you a probability estimate they give you a probability distribution for the probability! Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1] If you observe a coin flipped 100 times and it came up heads 65…

The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution. So everything in the paper is distribution and when you forecast for a binary ev…

Emphasizing this response. Bayesian models can always produce simple probabilities if you ask them to. E.g., given this data, what is the probability that the next flip is heads?

The fact that the model is represented as a distribution over Bernoulli parameter p doesn't contradict this: you just integrate over the posterior.

Re: The well-calibrated Bayesian [pdf] (1982)

#28

Post by glial »

danbruc wrote: post #22
glial wrote: post #17

Earlier quoted context omitted.

Someone walks out of a magic store holding a coin. They propose a bet. If they flip it 100 times and the proportion of heads is within [0.4, 0.6], you win $100. If it's not, you pay $100. Do you take that bet? Explanation: absent the magic store scenario, a `rational' person would take the bet. Your prior belief is that most coins are roughly unbiased. Given that they walked out of a magic store, you now have additio…

The bet does not really matter, the central question is whether they have a fair coin or if they are trying to me in some way. Even without the magic store, I would be very suspicious of anyone approaching random people with an offer like that. So I would certainly consider it likely, that they are trying to trick me. But the probability I would assign to this, would still be rooted in some frequency, somewhere under…

The key operation of Bayesian inference is integrating information. This can be from the same source (an additional coin flip, for example) or from different sources (coin flips, plus auxiliary knowledge about where the coin came from, or the person's motives).

Calculated frequency is a point-estimate of bias. A Bayesian estimate is a distribution of belief over possible values of the bias, integrating all available information.

Re: The well-calibrated Bayesian [pdf] (1982)

#29

Post by lioeters »

HPsquared wrote: post #4

Title a reference to The Well-Tempered Clavier?

I'm sure it is, and thought it's a clever and humorous title. It made me wonder, what other books and articles on scientific or mathematical subjects reference works of classical music, art, literature. Goedel Escher Bach, of course, but now I'm curious to find more of such interdisciplinary works.

Re: The well-calibrated Bayesian [pdf] (1982)

#30

Post by mitthrowaway2 »

wosk wrote: post #26
mitthrowaway2 wrote: post #25

Earlier quoted context omitted.

No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.

yes and in my example the ending point of the calculation is 0.58. I make an intermediate step explicit in this derivation because it is important for the understanding of the problem. This kind of intermediate step happens a lot in forecasting competitions, where participants are asked for their forecasts and their confidence in their forecasts. I want to show here that you need to include that in your forecast (all…

But again this seems like a TypeError.

If you're asking me for my forecast and my confidence in my forecast, I would say this:

Forecast: It will rain tomorrow. Confidence: 0.7

If you say "No no, I mean what is your confidence in the 0.7 number" then I have no idea what you're talking about. 0.7 is my confidence. It's not valid to attach a confidence to that number. When the next day comes, we simply find out whether it rains or does not rain, and whoever put the highest probability on the correct outcome wins.

If you're doing this with intermediate steps then I have to ask: where did the 0.7 probability come from as an intermediate step, and what is the epistemic meaning of the 0.8 attached to it? If you only felt 80% sure that you were 70% sure that it would rain tomorrow, then what is the proposition that you are 70% confident about?

Post reply on HN