Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.

Next step is explaining social science students the meaning of 'randomness' :)

Really, the amazing amount of bullshit social studies I have seen 'proven' by statistics. Amazing new insights like 'if children wear green shirts, while the teacher has a blue shirt the cognitive attention span is 12.3% higher than children wearing purple shirts. The effect was measured with a significance of bla bla bla.'

Software like SPSS facilitates this even more. People with no notion of random effects or probablilty theory click on the 'proof my research' button and even get it published.

So a lot more work in this area!

It's not just p-values. Some people just don't understand even very basic statistics.

I remember talking to one person in marketing who ran surveys of the company's users. They would send out a survey to all registered users, get back responses from 1% of them or something, and then proceed to report findings based on the responses. They were really happy, since a 1% response rate is great for surveys like this.

I tried to explain to them that all of this statistical machinery relies on having a random sample, and a self-selected sample is not that. No effect whatsoever. Surveys like this are standard practice in the industry. Why are you making trouble, geek-boy?

The idea is that you will decide when you've either expended as much energy as you are willing to, or when you're convinced that you won't make a different decision.

There is a simple symmetry argument that shows that the odds of a random walk reaching sqrt(N) in one direction and then getting to the opposite direction by N observations is exactly the same as the odds of a random walk reaching 2 sqrt(N) by the time you reach N observations. So the sqrt(N) rule is a simplified version of, "Call off the test when you've reached 95% certainty that we have our final answer."

You can easily make a million minor improvements. But I can't come up with a simpler rule meeting the business needs which is less likely to lead to dangerous misunderstandings.

"Decide when you pass this threshold, or when you've put out as much effort as you're willing to."

And the effort required to testing are easily understood as well.

"The cost of a bad A/B test is at most 100 lost sales and remembering it is there for 2 months." (This would be for an organization that expects 10,000 sales in 2 months, and is willing to leave tests up for that long.)

And if memory serves me (I'd have to redo the calculation) the rule of thumb in this case is that wins below 1% are called randomly, above 2% are usually called right, you're confident of getting the right answer above 5%, and even over many tests you are confident of never making a 10% mistake.

That last rule of thumb is important. Small organizations can't benefit from A/B tests. Large ones are stupid not to use it heavily. And very few people have a good sense of whether it makes sense for them.

hmm, what you are discussing above is typically called the formulation of the prior, not the calculation of it. And I agree that this formulation can be problematic. It is almost certainly the most contentious element in Bayesian inference.

However, I will note that your examples are largely overstating the problem. The prior is not typically as subjective as you are implying. Furthermore, the Bayesian prediction converges to the frequential result in the limit of 'large' data. If your results are so highly sensitive to your prior that the results drastically change, then you either have:

1) insufficient data, at which point the frequentist results would also certainly have been as bad or worse (the prior acts as a regularization) 2) a misspecified prior

Either way, it is absolutely required that you perform tests to ensure your results are not greatly sensitive to your choice of prior, or if so that this is clearly noted as a modeling assumption. And bayes naturally has many means to test precisely this, such as cross-validation, bayes-factors, hierarchical, etc.

I also note that this is why I typically prefer Jeffreys Priors in my work. I mention this so that one can see that not all priors are 'subjective'. These are difficult to use in some fields, admittedly.