P < 0.05 Considered Harmful

I find it interesting that the prior probability example always happens to be the one where the prior is correct, while the opposite direction is never given more than half a line. You can very easily turn it around so that an unreasonably large amount of evidence is needed to move a strongly believed wrong prior.

My favourite example is to do that with normal distributions and the posterior ends up as "let's meet in the middle and agree that we both were wrong" -a very low variance posterior with mean right in the middle between prior and data.

Not that fine. The author claims that neutral changes are not costly for users so the only reason to avoid them is to avoid wasting time/money. But that's not really true. Pointless churn annoys users. If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes that you thought would have benefit but which don't actually do anything because your threshold for something being considered significant is too loose.

I agree that the article is somewhat about the idea that having a Standard Default p-value Threshold is unwise, but I think it's mostly suggesting that in its particular context it's generally better to use a larger p-value. "Defaults matter", the author says (not "Defaults are a trap"). "Do we need that much risk aversion?" (not "Sometimes we need more risk aversion, sometimes less"). "If we were starting over ... I doubt that's what we would pick" (not "If we were starting over, I think we'd have a process that begins by considering what p-value threshold would be appropriate"). At the end, he considers situations where the criterion used is strict and "the company progresses too slowly" (but not ones where it's lax and "the company moves too fast and breaks too many things" or "the company thrashes about unstably") and "Why not just admit you want something akin to p=0.25 in the first place" (not "... something akin to p=0.25 some of the time, and something akin to p=0.001 some of the time?") and "I believe that nobody wants to write down a large p-value threshold, because it feels unscientific" (and nothing about reasons why people might be reluctant to write down an unusually small p-value threshold).

I agree, in case it needs to be said, that if you are doing hypothesis testing then you should adapt your choice of thresholds to your (explicit or implicit) prior. That was approximately the second of the three points I made.

"You should be choosing an example where reporting a p-value is more important than an effect size." Why should I be doing that? I think that almost always both are important (more precisely, I think that almost always thinking of what you've learned from an experiment into a p-value and a point-estimate effect size is suboptimal, but you want the information both of those things are trying to tell you). And if I've correctly understood what you say about rankings, I completely disagree with it; almost none of the time is the p-value what you should be ranking things on. Again, consider those two medications in my hypothetical example: I do not believe any reasonable person would think that the right way to rank them relative to one another matches the p-values. (For comparing two individual students? Yeah, maybe, kinda. But approximately 0% of cases where people compute p-values are analogous to that.)

I agree that picking any single metric and applying brainless black-and-white rules using it is liable to get you in trouble, whatever that metric is. But some metrics are better than others, and a fixed p-value threshold (1) has been a widespread common practice and (2) is for most purposes a really bad idea.

> Pointless churn annoys users.

Sure, excessive rate of change is bad. Even if it's all validated and well-proven.

> If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes

To measure P, I have to make the change.

There's plenty of situations where accumulating enough data to meet a p<0.05 threshold would take months. If my prior is that it's a reasonably good change and has a decent chance to be helpful, and then I take a measurement that makes that roughly 5x more likely... it's OK to push the button. There are many decisions in business that have to be made with much less information or statistical proof than this.

Yes; as I said, wasn't disagreeing per se, just clarifying important context more than anything.

> I completely disagree with it; almost none of the time is the p-value what you should be ranking things on.

Not necessarily true; people compare models on the basis of their p-values all the time; but in any case, that's not what I was referring to here. I was saying the p-value in itself expresses a ranking. That's just what a p-value is by definition: a ranking of the particular compatibility score of the data-event under consideration, with respect to the whole domain of compatibility scores resulting from all possible data-events considered.

And yes, when the score itself is more important than the ranking, it's prudent to focus on that; when the p-value "ranking" is more important, it's prudent to focus on that instead; ideally, people should consider both.

Having said that, the medical example is a weird one. There's a specific sentence there that I have a problem with:

> It's more uncertain because the sample size is smaller [...] I would definitely prefer the second drug.

Perhaps this is simply a misnomer, since one can only be so precise when using language, but as expressed above, I feel this is wrong, as it alludes to an expression of posterior probability, which is not information given to you by a p-value in this setting. More specifically you should be saying it's not "precise". The subtle difference being, you are interpreting this as a "despite the wide range, it's most likely to be around 0.1, so I'll take my chances", whereas what the confidence interval is really telling you is "the data would be atypical for models outside this large range, but not inside", meaning there is an almost equal chance of the drug harming you as there is giving you a benefit. So, no, I wouldn't prefer the 2nd drug; I'd actually prefer the 1st one which has a guaranteed, albeit smaller benefit. This is a very different statement from a credible interval which more or less would state that 0.1 is the most likely value.

So in fact, the p-value would be a better measure of comparison here as well. You're comparing the compatibility of two different data-events (which in this case reflects samplings) against the same (null) hypothesis; so you should prefer the event that is least compatible with the null (and ideally, you should propose an alternative hypothesis, and confirm that it demonstrates higher compatibility with that one as well). But having said this, comparing p-values typically comes in the reverse scenario, were one has a single data-event, and wants to compare two different hypotheses against the same data-event, since that gives information on which model is most compatible with what you observed.

Anyway. Sorry, I know you know, I'm just enjoying the exposition :)