Frequentists should more often consider using Bayesian methods
thestatsgeek.com
Frequentists should more often consider using Bayesian methods
1–10 of 39 posts
Re: Frequentists should more often consider using Bayesian methods
#2I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons."
In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for Bayesianism on their head.
I'm actually neither Bayesian nor frequentist, or both, depending on how you look at it, but I think Bayesianism is being overhyped. Sure, the prior is part of the model, but if you can estimate something without adding extra baggage, why not?
Imagine doing a meta-analysis, and now having all the extra heterogeneity due to priors. Why add that?
My guess is a lot of the appeal of Bayesianism has to do with the success of the machinery surrounding it, like MCMC, which is sort of automatic and has certain other appeals. As people realize you can do stochastic optimization with raw ML inference, some of the appeal will probably dissipate a bit (although not entirely).
Re: Frequentists should more often consider using Bayesian methods
#3This argument would seem to assume that a frequentist is required to use maximum likelihood estimation for all problems.
If you remove that assumption, the distinction between Bayesian and frequentist methods becomes murkier. If you ultimately want a point estimate, you don't particularly care whether a method constructs a posterior distribution as an intermediate step.
Re: Frequentists should more often consider using Bayesian methods
#4I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons." In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for…
> so then why does it matter?
I recommend reading this[1] essay which explains exactly the difference between the two methods. (I'm linking to part 3 because it specifically answers your questions, but the entire series is worth reading)
Why does it matter? Frequentist and Bayesian methods offer different interpretations of what your result means.
... speaking broadly, frequentists consider model parameters
to be fixed and data to be random, while Bayesians consider
model parameters to be random and data to be fixed.
> the frequentist and Bayesian methods are the same
...this isn't always true. See [1] for the details.
[1] http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...
Re: Frequentists should more often consider using Bayesian methods
#5...why does the order you do your math in have to be a part of your identity?
Re: Frequentists should more often consider using Bayesian methods
#6I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons." In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for…
If you are trying to answer the same question in the same way there isn't much if a difference.
Frequentist and bayesian statistics are different paradigms, not different number crunchers.
---
I talk about this with graphs and such in my blog post (https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...) but here's the short version:
Common question in our line of work: when should I end my A/B test? End too early and I have wrong conclusions, end too long and it's expensive. Classic exploration vs. exploitation.
Frequentist statistics will tell us to suck it up and just choose a sample size, or write down every time we look at the numbers to adjust for peeking effect, or assume the worst case that we always peek and wait a very long time.
Bayesian statistics tells us not to use a single termination condition, but adjust the samplings according to our continuously updating priors. (This is what Google Analytics does. https://support.google.com/analytics/answer/2844870) In fact, this turns out to be the mathematically proveable optimal solution for maximizing payoff. This latter approach is beyond the reach of frequentist methods because it's just a completely different paradigm.
The question of frequentist and bayesian not "which is more correct", but rather "which paradigm best matches the question I have?" If your testing paradigm is fixed sample sizes, choose frequentist. If it is iterative (like a lot of ML), choose bayesian.
Re: Frequentists should more often consider using Bayesian methods
#7I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons." In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for…
If the inferential question you're interested in is, "given data X, what do I conclude about underlying cause/variable/parameter T?", then you are a Bayesian, like it or not.* Sure you can define a likelihood function p(X|T), but that doesn't give you p(T|X) unless you multiply by a prior p(T).
Now certainly p(T|X) is not always the question, but in the vast majority of cases people do want to use data to draw conclusions, and will misinterpret likelihood-based confidence intervals as posterior credible intervals because the latter are what they intuitively wanted. By doing so they implicitly assume a flat prior regardless of whether that that is reasonable or even mathematically coherent for the problem in question.
The Bayesian argument is not that you have to use an informative prior (though often this can be very helpful!), but that since some sort of prior is mathematically necessary to answer the questions that people intuitively want statistics to answer, we should make that explicit and try to understand how the prior affects our conclusions, not just sweep it under the rug.
* If the question you're interested in is, "if I run some method in many repeated trials, how often will it identify the true parameter?" then you are a frequentist. I think it's much rarer for this to genuinely be someone's intuitive question, but it's certainly valid. And of course it's valid to ask both questions at once, in which case you might end up analyzing the frequentist properties of a Bayes-derived method.
Re: Frequentists should more often consider using Bayesian methods
#8... why does the order you do your math in have to be a part of your identity?
tl;dr LessWrong.
Re: Frequentists should more often consider using Bayesian methods
#9I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons." In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for…
If the inferential question you're interested in is, "given data X, what do I conclude about underlying cause/variable/parameter T?", then you are a Bayesian, like it or not.* Sure you can define a likelihood function p(X|T), but that doesn't give you p(T|X) unless you multiply by a prior p(T). Now certainly p(T|X) is not always the question, but in the vast majority of cases people do want to use data to draw conclu…
No, if you are drawing conclusions from only the data presented you are not doing Bayesian. Further, there are more than 2 options.
Re: Frequentists should more often consider using Bayesian methods
#10I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons." In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for…
Frequencist methods do not remove the prior, they merely hide it, make it implicit. Usually the implicit prior is reasonable but being blind of your prior still risks making you trip on subtle trade-offs and biases for more complex problems.