51–60 of 89 posts
Maybe tech industry insiders can tell me this ... but do real people actually make product decisions based solely on p < 0.05? Seems like the author is writing about a contrived problem.
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
I think if hypothesis testing is understood properly, these objections don't have much teeth. 1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.) 2. P-values control type I error. Well-powered designs control type I and type II error. Good control…
That’s why rust came along, to stop that behaviour, you simply can’t make that mistake, and hence the point is maybe there’s a better test to use than p value as a standard.
2σ is fine, but the benefit of modern technology is that we can tell exactly what standard deviation would be needed for the null hypothesis to randomly generate our results. Particle physics holds itself to an "industry standard" of 5 sigma, for example. The real conversation to be had is -- what standard deviation will we tolerate? Is this something we'll keep doing, and thus A/B test ourselves into a (perhaps quit…
It's 5σ for discovery, 2σ for null hypothesis "confirmation".
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…
If anything I wrote appeared to be saying "just look at effect size, it's the only thing that matters" then that was an error on my part. I definitely didn't intend to say that.
But I was responding to an article saying "pnever mentions effect sizes at all. I think that's enough to demonstrate that, in context, "it's bad to look at p-values and ignore effect sizes" is not in fact a straw man.
Incidentally, I am not convinced that the p-value as such is often a good way to assess how likely it is that your results are due to random chance. Suppose you see an effect size of 1 unit with p=0.05. OK, so there's a 5% chance of getting these results if the true effect size is zero. But you should also care what the chance is of getting these results if the true effect size is +0.1. (Maybe the distribution of errors is really weird and these results are very likely with a positive but much smaller effect size; then you have good evidence against the null hypothesis but very weak evidence for an effect size of the magnitude you measured.) In fact, what you really want to know is what the probability is for every possible effect size, because that gives you the likelihood ratios you can use to decide how likely you think any given effect size is after seeing the results. For sure, having the p-value is better than having nothing, but if you were going to pick one statistic to know in addition to (say) a point estimate of the effect size, it's not at all clear that the p-value is what you should choose.
Earlier quoted context omitted.
I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…
I agree that you shouldn't look only at effect sizes any more than you should look only at p-values. (What I would actually prefer you to do, where you can figure out a good way to do it, is to compute a posterior probability distribution and look at the whole distribution. Then you can look at its mean or median or mode or something to get a point estimate of effect size, you can look at how much of the distribution…
It would be cool if interfaces caught up so I could just draw a basic estimate of my prior and then see the posterior graph afterwards.
Earlier quoted context omitted.
I think if hypothesis testing is understood properly, these objections don't have much teeth. 1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.) 2. P-values control type I error. Well-powered designs control type I and type II error. Good control…
That sounds like if you write proper C code correctly you don’t make memory errors when in reality it’s very common to not write correct code. That’s why rust came along, to stop that behaviour, you simply can’t make that mistake, and hence the point is maybe there’s a better test to use than p value as a standard.
Earlier quoted context omitted.
Thanks for that. I'll give [1] a read. I'm familiar with [2], and cited one of those papers in the blog. About the stupid or harmful nature of null hypothesis testing in general, what do you recommend instead for decision making and for summarization of uncertainty? In the scenario of large (yet fast moving) organizations where most people will have little stats background.
Thanks, I hope you find Gigerenzer useful. The paper is a bit academic, but he also wrote a couple of nice popular science books on the (mis-)perception of numbers and statistics, those might be useful in a business environment. For real-world applications outside engineering and academia, I would rely heavily on confidence intervals and/or confidence bands. For example, the packages from easystats [1] in R have quit…
Would the policy be "look at our confidence intervals later and then decide what to do"? One remaining issue is how to have consistent decision criteria, and to convey it ahead of time. Imagine a context with 10-50 teams at a company that run experiments, where the teams are implicitly incentivized to find ways to report their experiments as successful. Quantified criteria can be helpful in minimizing that bad incentive.
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…
The only question is… would you rather be up front and explicit about your assumptions, or not?
An uninformative prior is an assumption, even if it’s the one that doesn’t bias the posterior (note that here “bias” is not a bad word).
Earlier quoted context omitted.
I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…
You can always encode the prior. If you take the frequentist approach and ignore bayesian concepts, it’s the same as just going bayesian but with an “uninformative prior” (constant distribution). The only question is… would you rather be up front and explicit about your assumptions, or not? An uninformative prior is an assumption, even if it’s the one that doesn’t bias the posterior (note that here “bias” is not a ba…
I'm also not sure why everyone perceived my comment as anti-bayesian.
Maybe tech industry insiders can tell me this ... but do real people actually make product decisions based solely on p < 0.05? Seems like the author is writing about a contrived problem.
The "stasis" and "arbitrarily adjustments" regimes that I wrote about are certainly ones that I've seen, which don't rely solely on p But let me turn that around and ask: what product decision regime do you see most often or think would be the most relevant to use as an example? I'd be happy to hear your perspective and make sure I keep it in mind for future blogs.
Anyway when you say "Furthermore, it's not only about whether 0.05 is the sole criteria, but also about whether it's a useful criteria for us to highlight at all, depending on whether the anchoring effect of it is damaging relative to alternatives." I love this analogy of the null hypothesis vs. alternative hypotheses in frequentist statistics to the 'anchoring effect' cognitive bias that you try to work to your advantage in marketing and sales and negotiation or management. https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)
If you don't want to be tied to p-values and you only care about downstream decisions rather than quantifying beliefs, you can use some ideas in decision theory https://en.wikipedia.org/wiki/Decision_theory
For example, maybe you are deciding between two alternative ways of doing something. You don't know which one is better, and you are confronted with not only the decision of which one to use, but also with the decision of whether to experiment (with A/B testing for example) to be more sure of which one is right, versus whether to exploit the one that you currently think is better. This is the multi-arm bandit problem and it doesn't necessarily use p-values so your intuition is right! https://en.wikipedia.org/wiki/Multi-armed_bandit
Maybe that's not your situation. Maybe your situation is that you have an existing business process and you want to know whether to switch to one that might be better. Someone might say it's a p-value problem, but again I agree with your intuition that it really isn't best to think about it that way, especially when there is a cost to switching. Instead, it's a more complicated decision that depends on what is the switching cost, how much better you think the new process would be (including uncertainty of it), and what kind of business horizon you care about. There might even be a multi-armed bandit effect again even in this situation, where you also have to weigh the costs of reducing your uncertainty of the switching improvement or even of reducing the uncertainty of the switching cost itself.
Anyway, these problems do involve concepts from probability and statistics but it's for sure true that the decisions don't always reduce to P < 0.05 at the end! Good luck best wishes living your best techbro life!