What Is Bayesian/Frequentist Inference? (2012)

This blog is by Larry Wasserman, so i think his advice should be taken seriously. I agree that there are uses of both philosophies, and that statisticians should be pragmatic rather than dogmatic.

My issue is that his advice is most useful for statisticians working in the abstract, but it doesn’t really help people working with real data. Scientists and data analysts just want to know how to analyze their data, and this does not help them. I know that stats isn’t a cookbook, but we could put some guard rails down that help practitioners with their problems.

This falls apart in higher dimensions, but in the example given in the article the two answers only differ because they have different priors. If you repeat the bayesian analysis using the prior \theta ~ N(0, x), and let x go to infinity, then you approach the frequentist answer.

In my opinion, https://stats.stackexchange.com/questions/2272/whats-the-dif... is a better explanation of the difference between confidence and credible intervals.

Editing to add more commentary to my link:

If you've read the link, one of the principal objections of the frequentist is "What if the jar is type B? Then your interval will be wrong 80% of the time, and only correct 20% of the time!"

This is because, if you look at the original numbers, jar B has all types of cookies, and therefore any single draw from jar B can "look like" any other jar, and because other jars have more concentrated cookie types, they are "more likely" answers for each potential sample.

This issue also comes up with the frequentist analysis! If you look at the confidence intervals, they all say "This jar could be jar B". Instead of being bad at detecting jar B, they are good at always considering jar B no matter the evidence – but it's the same uncertainty.

The bayesian version of this objection is "when you pull a cookie with 3 chips your interval is only correct 41% of the time". This is because if we get jar A, we'll probably draw a 2-chip cookie, so it's outside of our confidence interval.

But note that we probably don't really care about the confidence interval or credibility interval. It's basically a hack to take a probabilistic problem and turn our answer into black and white. To say "these hypotheses are valid and these are invalid".

But this is statistics! If you just take the Bayesian approach, and throw out the need to create an arbitrary interval, you can just stop at the table titled P(Jar|Chips). That's all the information you need. If you draw a N-chip cookie, you can use that table to update your P(Chips_2) for a second draw, and you'll get a concrete probabilistic answer. Yes, you have to assume a prior. But frequentist statistics literally can't answer this question! Without a prior, there's no way to turn P(Chips | Jar) into a P(Jar | Chips) to update on, so you can't track your evidence to get better predictions. You just sit there saying "well, my interval meets the criterion even in the worst case".

Whilst successful in my career and user of probability, statistics, and inference on a regular basis, I simply cannot understand what's being discussed here.

I don't even want to understand it. Just like quantum, half the argument seems to be the a mismatch between mental models and actual reality.

Guardrails for stats are something that I've put a lot of thought towards. The fundamental challenge is that statistics is operating in a world of incomplete information. Statistical measures are almost never monotone with respect to new information, and so any new piece of context might completely invalidate an analysis. Going beyond the abstract requires an intimate knowledge of the domain being analyzed, and the limitations of statistical methods as applied to that domain. "Guardrails for stats" have to be domain- and even dataset-specific.

In the end I believe the Bayesian inference is more straightforward to implement and understand if you can afford computationally sampling of the posterior. So I think at least in physics there is a shift towards Bayesian approaches.