Maybe it's because my formal math training is not in probability and statistics, but it's so bizarre to me that in a technical situation people would let a philosophical position dictate their approach rather than best tools for the job. Sometimes I'll solve a math problem analytically, and sometimes its easier to do it numerically. But it would be foolish for me to take a hardline stance on one vs the other. Rather…
How do you decide which tool is the best for the job?
Frequentists should more often consider using Bayesian methods
21–30 of 39 posts
Re: Frequentists should more often consider using Bayesian methods
#22Re: Frequentists should more often consider using Bayesian methods
#23Maybe it's because my formal math training is not in probability and statistics, but it's so bizarre to me that in a technical situation people would let a philosophical position dictate their approach rather than best tools for the job. Sometimes I'll solve a math problem analytically, and sometimes its easier to do it numerically. But it would be foolish for me to take a hardline stance on one vs the other. Rather…
Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all possible conclusions. No matter how you approach the problem, applying probability theory correctly will yield the same answer.
The only drawback is that doing it correctly is often computationally intractable. So we take shortcuts, such as Monte Carlo, hence approximating the correct method, possibly yielding different results depending on the exact nature of the approximation.
Frequentist methods however can yield different conclusions depending on who you approach the problem, before you even approximate anything. In part because it sometimes takes into account irrelevant information, such as the researcher's state of mind at the time of the experiment (did I stop at 100 trials because I just got statistical significance, or did I stop at 100 trials because that was decided in advance?).
To me, such insanity is hard to fathom. Are frequentist methods that stupid? The glimpse I have got so far indicate they might be.
Re: Frequentists should more often consider using Bayesian methods
#24Earlier quoted context omitted.
tl;dr LessWrong.
Yudkowsky has done such a stellar job in making some fairly simple ideas look like a weird cult.
Of course it will sound like a weird cult. I'm not sure why anybody would have any problem with that.
Re: Frequentists should more often consider using Bayesian methods
#25Re: Frequentists should more often consider using Bayesian methods
#26What are the Bayesian methods I would use if I have multiple related outcomes and multiple predictors...you know...the situation in which many scientists find themselves with longitudinal data and typically turn to repeated measures and random effect general linear models?
Re: Frequentists should more often consider using Bayesian methods
#27Earlier quoted context omitted.
If you are trying to answer the same question in the same way there isn't much if a difference. Frequentist and bayesian statistics are different paradigms , not different number crunchers. --- I talk about this with graphs and such in my blog post ( https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... ) but here's the short version: Common question in our line of work: when should I end my A/B test? End t…
Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data[1]. Or, if you want to get fancy you could use a multi-armed bandit approach which is probably optimal in many situations in perhaps a more robust way than many Bayesian methods[2]. Really both can work well. My advice is, use whichever you know well enough to utilize effectiv…
And, as he said, you have to make adjustments to account for these interim analysis.
Re: Frequentists should more often consider using Bayesian methods
#28Earlier quoted context omitted.
No, if you are drawing conclusions from only the data presented you are not doing Bayesian. Further, there are more than 2 options.
Have you heard of the universal prior? It allows you to do exactly that with Bayesian statistics. Although it is so esoteric that one might argue that it is more the field of algorithmic statistics which merely employs Bayesian. https://en.m.wikipedia.org/wiki/Algorithmic_probability https://en.m.wikipedia.org/wiki/Solomonoff%27s_theory_of_ind...
Re: Frequentists should more often consider using Bayesian methods
#29Earlier quoted context omitted.
No, if you are drawing conclusions from only the data presented you are not doing Bayesian. Further, there are more than 2 options.
you can draw conclusions from only the data presented to you doing bayesian inference.
Supose you don't know what color a car is, so you look and see it's green and then you look again and still green. Now, what's the Bayesian calculation.
Well how accurate are your eyes, how stable is car color ect.
Re: Frequentists should more often consider using Bayesian methods
#30Maybe it's because my formal math training is not in probability and statistics, but it's so bizarre to me that in a technical situation people would let a philosophical position dictate their approach rather than best tools for the job. Sometimes I'll solve a math problem analytically, and sometimes its easier to do it numerically. But it would be foolish for me to take a hardline stance on one vs the other. Rather…
This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best method is merely the easiest to apply, or the one that'll yield the answer fastest. Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all p…
I've got slides that illustrate exactly your example. Look at the graph below:
https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2...
Your different conditions simply represent the probability of the blue line crossing two different other lines.
Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.