Earlier quoted context omitted.
No, if you are drawing conclusions from only the data presented you are not doing Bayesian. Further, there are more than 2 options.
Have you heard of the universal prior? It allows you to do exactly that with Bayesian statistics. Although it is so esoteric that one might argue that it is more the field of algorithmic statistics which merely employs Bayesian. https://en.m.wikipedia.org/wiki/Algorithmic_probability https://en.m.wikipedia.org/wiki/Solomonoff%27s_theory_of_ind...
Frequentists should more often consider using Bayesian methods
31–39 of 39 posts
Re: Frequentists should more often consider using Bayesian methods
#32Earlier quoted context omitted.
Yudkowsky has done such a stellar job in making some fairly simple ideas look like a weird cult.
He explained stuff plainly, using references (sci-fi, japanese manga and anime) that are popular among a fairy restricted set of people. It's not academic, and it's not all-inclusive. Of course it will sound like a weird cult. I'm not sure why anybody would have any problem with that.
In my case, an IRC channel that I previously enjoyed (relationship-related) was unofficially absorbed into LessWrong's network of channels. This resulted in people treating it as a meat market despite being asked not to, then a debate platform despite being asked not to, some refused the existence of trans people to their faces because they couldn't logic it out, we had one argue in favour of cheating in a channel that explicitly mentioned openness and honesty in the topic, they damn near had a riot if we ever dared ban one of them for being spectacularly awful, and they generally put the value of other people's emotions at close to 0 in their messed-up logic for interacting with the world. This carried on for months. Apparently after I left most of them were finally banned, though not before a good many of the less awful people in the channel left for good. This was the first major issue the channel had in four years of its existence.
More generally, many people don't enjoy conflict, and the LessWrong way of discussing anything seems to be to turn it into a two-sided debate. Even when you're trying to discuss who you think you are, turning it into you defending yourself from attack. The concept of having a constructive conversation, seems to have entirely slipped past them, never mind the idea that not everything has to be debated right then and there.
Unfortunately, this isn't just my experience - various others I've spoken to have come across exactly the same sort of thing. Hence the extreme dislike from some people. I'm not sure whether the community explicitly pushes this way of interacting with the world, or whether it just attracts assholes and gives them a "logical" reason to be assholes, though.
Re: Frequentists should more often consider using Bayesian methods
#33Earlier quoted context omitted.
This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best method is merely the easiest to apply, or the one that'll yield the answer fastest. Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all p…
Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently . I've got slides that illustrate exactly your example. Look at the graph below: https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2... Your different conditions simply represent the probability of the blue line crossing two different other lines. Frequentism isn't…
That's exactly what I had in mind. It amounts to the same, really.
Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does.
Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid.
> Frequentism isn't stupid, but you do need to be really smart to use them correctly.
I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste.
When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them.
I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think?
> [VWO slides]
Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.
Re: Frequentists should more often consider using Bayesian methods
#34Earlier quoted context omitted.
Yudkowsky has done such a stellar job in making some fairly simple ideas look like a weird cult.
He explained stuff plainly, using references (sci-fi, japanese manga and anime) that are popular among a fairy restricted set of people. It's not academic, and it's not all-inclusive. Of course it will sound like a weird cult. I'm not sure why anybody would have any problem with that.
Re: Frequentists should more often consider using Bayesian methods
#35Earlier quoted context omitted.
If you are trying to answer the same question in the same way there isn't much if a difference. Frequentist and bayesian statistics are different paradigms , not different number crunchers. --- I talk about this with graphs and such in my blog post ( https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of... ) but here's the short version: Common question in our line of work: when should I end my A/B test? End t…
Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data[1]. Or, if you want to get fancy you could use a multi-armed bandit approach which is probably optimal in many situations in perhaps a more robust way than many Bayesian methods[2]. Really both can work well. My advice is, use whichever you know well enough to utilize effectiv…
---
Agreed 100% about multi-armed bandit which is what I was referring to. And the canonical solutions are in fact Bayesian :) See the Google Analytics link or lookup "Thompson sampling"
From your Wikipedia link:
"Probability matching strategies are also known as Thompson sampling or Bayesian Bandits, and surprisingly easy to implement if you can sample from the posterior for the mean value of each alternative."
Re: Frequentists should more often consider using Bayesian methods
#36Earlier quoted context omitted.
He explained stuff plainly, using references (sci-fi, japanese manga and anime) that are popular among a fairy restricted set of people. It's not academic, and it's not all-inclusive. Of course it will sound like a weird cult. I'm not sure why anybody would have any problem with that.
Mostly that some of us have had very poor experiences with them while trying to mind our own business. It's like living next to a cult's compound, and therefore having more concrete bad experiences with them than someone in the next city. In my case, an IRC channel that I previously enjoyed (relationship-related) was unofficially absorbed into LessWrong's network of channels. This resulted in people treating it as a…
I may even have been part of the problem for a while, posting semi-relevant LW links all over the place. I have since learned to focus my arguments into something that doesn't require having read the sequences. Maybe no longer frequenting LW helped.
Re: Frequentists should more often consider using Bayesian methods
#37Earlier quoted context omitted.
How do you decide which tool is the best for the job?
The one that yields the simpler solution.
Re: Frequentists should more often consider using Bayesian methods
#38What are the Bayesian methods I would use if I have multiple related outcomes and multiple predictors...you know...the situation in which many scientists find themselves with longitudinal data and typically turn to repeated measures and random effect general linear models?
Maybe this helps: https://arxiv.org/abs/1506.06201
Re: Frequentists should more often consider using Bayesian methods
#39Earlier quoted context omitted.
Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data[1]. Or, if you want to get fancy you could use a multi-armed bandit approach which is probably optimal in many situations in perhaps a more robust way than many Bayesian methods[2]. Really both can work well. My advice is, use whichever you know well enough to utilize effectiv…
Right, as I said, it can be done with frequentist statistics. This is what Optimizely does ( http://pages.optimizely.com/rs/optimizely/images/stats_engin... ). But (1) it is not simple and (2) it is not optimal. --- Agreed 100% about multi-armed bandit which is what I was referring to. And the canonical solutions are in fact Bayesian :) See the Google Analytics link or lookup "Thompson sampling" From your Wikipedia l…