Interpretation of confidence intervals and Bayesian credible intervals • Hacker News Board

Re: Interpretation of confidence intervals and Bayesian credible intervals

#11

Post by mjburgess » Sat, Nov 21, 2020, 4:20 PM UTC

maire wrote: post #5

alexpetralia wrote: post #2

It appears to me that the reason Bayesian probability is somewhat elusive is due to its metaphysical underpinning: what do we mean by probability? If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty. Then, what is probability? If everything will happen with 100% certainty, but probability certa…

> If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty. I don't know who "the many" are - but I thought determinism had already been disproved. I am not a physicist so I will not go into quantum mechanics - but I will take a simple example from Science Fiction, and that is the Temporal Paradox.…

How does this disprove determinism?

Re: Interpretation of confidence intervals and Bayesian credible intervals

#12

Post by mjburgess » Sat, Nov 21, 2020, 4:23 PM UTC

cultus wrote: post #8

alexpetralia wrote: post #2

It appears to me that the reason Bayesian probability is somewhat elusive is due to its metaphysical underpinning: what do we mean by probability? If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty. Then, what is probability? If everything will happen with 100% certainty, but probability certa…

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

I don't think you're engaging in the specific sense of objective (, subjective) meant here.

A frequentist interpretation of probability is objective in the sense that it grounds the probability value in objective features of the world.

A bayesian is subjective in that the probability valuation of an event is grounded in the belief-state of its observer.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#13

Post by analog31 » Sat, Nov 21, 2020, 4:26 PM UTC

cultus wrote: post #8

alexpetralia wrote: post #2

It appears to me that the reason Bayesian probability is somewhat elusive is due to its metaphysical underpinning: what do we mean by probability? If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty. Then, what is probability? If everything will happen with 100% certainty, but probability certa…

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

I think a further problem may be that probability could take on different meanings under different applications. I admit that I may be influenced by how I learned statistics -- in a math class that was primarily focused on proofs and not applications. But I've formed the view that math is math, we choose a math technique that works for the situation at hand, then we choose an interpretation that works for guiding and explaining what we're trying to do.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#14

Post by analog31 » Sat, Nov 21, 2020, 4:28 PM UTC

radford-neal wrote: post #9

cultus wrote: post #8

Earlier quoted context omitted.

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices. Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns. [0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

No, frequentist and Bayesian statistics are not equivalent. There are some special cases in which a frequentist 95% confidence interval and a Bayesian 95% credible interval based on some sort of default prior are numerically the same, but that doesn't happen in general. Statisticians would hardly have been vigorously debating the issue for two centuries if it didn't really matter.

From the article:

>>>> Let’s now suppose that we’ve done a Bayesian analysis. We’ve specified a prior distribution for the parameter, based on prior evidence, our subjective beliefs about the value of the parameter, or perhaps we used a default ‘non-informative’ prior built into our software package.

At first blush the difference is that the Bayesian is using more information. Now don't get me wrong, if Bayes theorem and its progeny give us useful tools for incorporating that information in our analyses, so much the better.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#15

Post by 6gvONxR4sf7o » Sat, Nov 21, 2020, 4:37 PM UTC

yccs27 wrote: post #7

There seems to be a nice duality between Bayesian and Frequentist inference [1]: Assume that both the system state and the observation are drawn from some joint probability distribution. There is some function γ of the system state which we seek to estimate. The experimentator applies some decision procedure d to the observation to get their result. A Frequentist will analyze the situation by conditioning on the the…

> Unbiasedness of the decision procedure means that γ is indeed the function it estimates the best, and it is not a better estimator for any other function γ'(θ).

Are you suggesting that unbiased estimators are necessarily better than biased ones? If so, check out Stein’s phenomenon for a counterexample. It’s common for biased estimators to dominate unbiased ones in terms of error rates. That’s where the bias variance trade off in ML comes from.

https://en.wikipedia.org/wiki/Stein's_example

Re: Interpretation of confidence intervals and Bayesian credible intervals

#16

Post by outlace » Sat, Nov 21, 2020, 4:51 PM UTC

KKPMW wrote: post #4

outlace wrote: post #3

Earlier quoted context omitted.

It seems a Bayesian interpretation of probability is more general. The frequency of events over an infinite number of trials is one way of interpreting probability for things that are able to be repeated. But this wouldn’t make sense to apply for an election that is only going to happen once and yet one still wants to be able to quantify uncertainty in these situations.

There are a few things intermingled in this election example. 1. The outcome of the election here is not a probability. It is the population value - the ratio of people voting for candidate X on the election date. It doesn't have to be repeated in the same way measurements of height for all people in United States would not have to be repeated, if instead of vote we were measuring heights. 2. Frequentist probability…

If you’re trying to quantify your uncertainty about who will win the election, a poll would only be part of it. You want to be able to combine disparate sources of information. Maybe there is preference falsification and you want to incorporate as some sort of prior. As things get further from simple sampling from a population the frequency interpretation makes less and less sense to me.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#17

Post by 6gvONxR4sf7o » Sat, Nov 21, 2020, 5:07 PM UTC

alexpetralia wrote: post #2

It appears to me that the reason Bayesian probability is somewhat elusive is due to its metaphysical underpinning: what do we mean by probability? If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty. Then, what is probability? If everything will happen with 100% certainty, but probability certa…

You’re correct that probability still works in a hypothesized deterministic universe. It’s a point that’s often too often forgotten, causing discussions to go down unnecessary rabbit holes debating foundations of quantum mechanics when discussing the roll of a six sided die.

Statisticians and mathematicians have gone very far down the path you’re discussing, and you might be interested in some sets of axioms that have come up around probability and relaxations of true/false logic.

The Kolmogorov axioms [0] are the “standard” probability axioms, and are phrased in terms of set theory and measure theory (not requiring any mention of physics or a physical universe!).

There are other ways to quantify degree of belief, however, and they are very interesting. Apparently Cox’s theorem [1] justifies a popular probability framework for Bayesians. But there are many more interesting ways to do degree of belief, like Dempster-Schafer theory [2], which I understand to be a plausibility calculus.

Everybody seems to find a single system and decide it’s the only one out there,

[0] https://en.wikipedia.org/wiki/Probability_axioms

[1] https://en.wikipedia.org/wiki/Cox's_theorem

[2] https://en.wikipedia.org/wiki/Dempster%E2%80%93Shafer_theory

Re: Interpretation of confidence intervals and Bayesian credible intervals

#18

Post by KKPMW » Sat, Nov 21, 2020, 5:23 PM UTC

outlace wrote: post #16

KKPMW wrote: post #4

Earlier quoted context omitted.

There are a few things intermingled in this election example. 1. The outcome of the election here is not a probability. It is the population value - the ratio of people voting for candidate X on the election date. It doesn't have to be repeated in the same way measurements of height for all people in United States would not have to be repeated, if instead of vote we were measuring heights. 2. Frequentist probability…

If you’re trying to quantify your uncertainty about who will win the election, a poll would only be part of it. You want to be able to combine disparate sources of information. Maybe there is preference falsification and you want to incorporate as some sort of prior. As things get further from simple sampling from a population the frequency interpretation makes less and less sense to me.

Sorry, I don't see how this reply is related to the original objection that frequentist methods are not applicable to events that happen once.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#19

Post by 6gvONxR4sf7o » Sat, Nov 21, 2020, 5:42 PM UTC

Confidence intervals are weird because of their very minimal definition. My favorite confidence interval procedure for iid data demonstrates why you need a tighter definition for a useful interval.

For a 93.75% confidence interval, draw 5 points (iid). If the last four are all greater than the first one, your CI is the whole real number line, otherwise it’s the empty set.

Once you draw some actual data and get a specific interval, you want to ask about some degree of belief that your specific interval contains the actual parameter. In the case that your CI is all numbers, you know for a fact that it contains the true parameter value. In the case that your CI is the empty set, you know for a fact that it doesn’t contain the true parameter value.

I like this CI procedure because it demonstrates two things. 1) The kind of reasoning for going forward from an unknown parameter to a random interval is very different than what you have working backwards from a specific interval back to the parameter. That asymmetry can be WEIRD. 2) The weirdness is possible if you limit yourself to only the CI definition, meaning that if you want it to be useful, you need something that rules out weird shit like my example.

The properties of specific CI procedures people actually use are generally much much better than what is allowed by the definition of a CI. If you want useful reasoning backwards from the interval, don’t try to reason solely from the definition of a CI.

Re: Interpretation of confidence intervals and Bayesian credible intervals

#20

Post by yccs27 » Sat, Nov 21, 2020, 6:16 PM UTC

6gvONxR4sf7o wrote: post #15

yccs27 wrote: post #7

There seems to be a nice duality between Bayesian and Frequentist inference [1]: Assume that both the system state and the observation are drawn from some joint probability distribution. There is some function γ of the system state which we seek to estimate. The experimentator applies some decision procedure d to the observation to get their result. A Frequentist will analyze the situation by conditioning on the the…

> Unbiasedness of the decision procedure means that γ is indeed the function it estimates the best, and it is not a better estimator for any other function γ'(θ). Are you suggesting that unbiased estimators are necessarily better than biased ones? If so, check out Stein’s phenomenon for a counterexample. It’s common for biased estimators to dominate unbiased ones in terms of error rates. That’s where the bias varianc…

Indeed, the unbiasedness condition for an estimator says nothing about its quality relative to other estimators. Instead, it requires that you cannot change the target function γ without the expected loss becoming worse.

The paper I referenced even includes a theorem saying that, as long as the value to be estimated can not be exactly deduced from the observation, no estimator is both Bayes-optimal and unbiased. However for many observations, the Bayes-optimal estimator becomes asymptotically unbiased.