Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interpretation of confidence intervals and Bayesian credible intervals

thestatsgeek.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Interpretation of confidence intervals and Bayesian credible intervals

#21

Post by hexane360 »

6gvONxR4sf7o wrote: post #19

Confidence intervals are weird because of their very minimal definition. My favorite confidence interval procedure for iid data demonstrates why you need a tighter definition for a useful interval. For a 93.75% confidence interval, draw 5 points (iid). If the last four are all greater than the first one, your CI is the whole real number line, otherwise it’s the empty set. Once you draw some actual data and get a spec…

I'm having trouble understanding your example. What value is the 93.75% confidence interval for? Is it for the population mean? If so, why does the sample order influence the result?

Re: Interpretation of confidence intervals and Bayesian credible intervals

#22

Post by 6gvONxR4sf7o »

hexane360 wrote: post #21
6gvONxR4sf7o wrote: post #19

Confidence intervals are weird because of their very minimal definition. My favorite confidence interval procedure for iid data demonstrates why you need a tighter definition for a useful interval. For a 93.75% confidence interval, draw 5 points (iid). If the last four are all greater than the first one, your CI is the whole real number line, otherwise it’s the empty set. Once you draw some actual data and get a spec…

I'm having trouble understanding your example. What value is the 93.75% confidence interval for ? Is it for the population mean? If so, why does the sample order influence the result?

An x% confidence interval takes data D and produces an interval i. For some data generating process from parameter m to a random dataset D (random like if you do the same experiment again, you’ll get different data), the probability that i(D) contains m is x%. That’s the definition of an x% CI.

In my example i(D) is a function of the data (a function of the ordering), and D is a random dataset. Since it’s iid by assumption (sneakily also assuming the probability of an exact repeat is zero), the probability that the interval contains all numbers is 93.75% (1-1/2^4). Otherwise it’s the empty set.

Unpacking that, suppose you have a real number m. The probability that i(D) will contain m (with D as the random variable) is 93.75%, so it is a valid confidence interval for m.

m could be the population mean, the population median, your dog’s age, whatever. The interval depends on the data, but not on the parameter, and the definition of a CI says that’s fine.

It’s a demonstration that definition of a CI alone isn’t really useful for reasoning about a parameter given an interval. You need to know more about the specific data generating process and function i that led to it in order to make sure it’s useful.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#23

Post by zmmmmm »

6gvONxR4sf7o wrote: post #22
hexane360 wrote: post #21

Earlier quoted context omitted.

I'm having trouble understanding your example. What value is the 93.75% confidence interval for ? Is it for the population mean? If so, why does the sample order influence the result?

An x% confidence interval takes data D and produces an interval i . For some data generating process from parameter m to a random dataset D (random like if you do the same experiment again, you’ll get different data), the probability that i(D) contains m is x%. That’s the definition of an x% CI. In my example i(D) is a function of the data (a function of the ordering), and D is a random dataset. Since it’s iid by ass…

I'm confused by your example also ... if the points are iid then how can the order influence the estimate? They are independent of each other, so there is no way for points 1 - 5 to influence the next data value sampled from the distribution.

Or to put it another way, if i(D) is a function of the ordering, then isn't by definition the ultimate random process observed through i(D) not iid even if D is iid?

Re: Interpretation of confidence intervals and Bayesian credible intervals

#24

Post by 08-15 »

radford-neal wrote: post #9
cultus wrote: post #8

Earlier quoted context omitted.

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

No, frequentist and Bayesian statistics are not equivalent. There are some special cases in which a frequentist 95% confidence interval and a Bayesian 95% credible interval based on some sort of default prior are numerically the same, but that doesn't happen in general. Statisticians would hardly have been vigorously debating the issue for two centuries if it didn't really matter.

It's been only one century since Ronald Fisher misunderstood statistics and thereby created what's now known as 'frequentist statistics'.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#25

Post by tpoacher »

6gvONxR4sf7o wrote: post #22
hexane360 wrote: post #21

Earlier quoted context omitted.

I'm having trouble understanding your example. What value is the 93.75% confidence interval for ? Is it for the population mean? If so, why does the sample order influence the result?

An x% confidence interval takes data D and produces an interval i . For some data generating process from parameter m to a random dataset D (random like if you do the same experiment again, you’ll get different data), the probability that i(D) contains m is x%. That’s the definition of an x% CI. In my example i(D) is a function of the data (a function of the ordering), and D is a random dataset. Since it’s iid by ass…

that is not a correct interpretation of a confidence interval. what you describe refers to a posterior probability, which is not what CIs do. See Greenland et al for a good paper on this.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#26

Post by cultus »

radford-neal wrote: post #9
cultus wrote: post #8

Earlier quoted context omitted.

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

No, frequentist and Bayesian statistics are not equivalent. There are some special cases in which a frequentist 95% confidence interval and a Bayesian 95% credible interval based on some sort of default prior are numerically the same, but that doesn't happen in general. Statisticians would hardly have been vigorously debating the issue for two centuries if it didn't really matter.

Every frequentist technique has a Bayesian interpretation and vice-versa. Confidence intervals are equivalent to credible intervals with certain priors.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#27

Post by cultus »

mjburgess wrote: post #12
cultus wrote: post #8

Earlier quoted context omitted.

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

I don't think you're engaging in the specific sense of objective (, subjective) meant here. A frequentist interpretation of probability is objective in the sense that it grounds the probability value in objective features of the world. A bayesian is subjective in that the probability valuation of an event is grounded in the belief-state of its observer.

My point is you that you cannot ever actually claim to ground a probability value in objective features of the world. Many events aren't repeated, for example.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#28

Post by cultus »

analog31 wrote: post #13
cultus wrote: post #8

Earlier quoted context omitted.

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

I think a further problem may be that probability could take on different meanings under different applications. I admit that I may be influenced by how I learned statistics -- in a math class that was primarily focused on proofs and not applications. But I've formed the view that math is math, we choose a math technique that works for the situation at hand, then we choose an interpretation that works for guiding and…

It's just a perspective, but Bayesian inference makes the prior explicit, wheareas it is implicit with frequentist inference. You can't not have a prior.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#29

Post by 6gvONxR4sf7o »

tpoacher wrote: post #25
6gvONxR4sf7o wrote: post #22

Earlier quoted context omitted.

An x% confidence interval takes data D and produces an interval i . For some data generating process from parameter m to a random dataset D (random like if you do the same experiment again, you’ll get different data), the probability that i(D) contains m is x%. That’s the definition of an x% CI. In my example i(D) is a function of the data (a function of the ordering), and D is a random dataset. Since it’s iid by ass…

that is not a correct interpretation of a confidence interval. what you describe refers to a posterior probability, which is not what CIs do. See Greenland et al for a good paper on this.

It is most definitely not a posterior probability, but it’s always hard to tersely write out which process you’re describing in plain English and no formalism. All the probabilities I mention are probabilities in the sense of a CI. And I’m too lazy to write it out thoroughly.

This paper describes the situation more thoroughly

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.691...

Re: Interpretation of confidence intervals and Bayesian credible intervals

#30

Post by 6gvONxR4sf7o »

zmmmmm wrote: post #23
6gvONxR4sf7o wrote: post #22

Earlier quoted context omitted.

An x% confidence interval takes data D and produces an interval i . For some data generating process from parameter m to a random dataset D (random like if you do the same experiment again, you’ll get different data), the probability that i(D) contains m is x%. That’s the definition of an x% CI. In my example i(D) is a function of the data (a function of the ordering), and D is a random dataset. Since it’s iid by ass…

I'm confused by your example also ... if the points are iid then how can the order influence the estimate? They are independent of each other, so there is no way for points 1 - 5 to influence the next data value sampled from the distribution. Or to put it another way, if i(D) is a function of the ordering, then isn't by definition the ultimate random process observed through i(D) not iid even if D is iid?

If the points are iid, it’s crazy for the order to affect the estimate, but not off limits. This paper goes into it more thoroughly: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.691...
Post reply on HN