21–30 of 89 posts
Earlier quoted context omitted.
Xcom isn’t a good example because the game actively lies to you with displayed probabilities https://youtu.be/l0KEDYFWbVc
Can you elaborate on how XCOM lies? I often suspected this (but you can never be sure, since human intuition is bad at probabilities). Is there hard evidence?
But I want to recommend the wonderful paper "the null ritual" [1] by Gerd Gigerenzer et al. It shows that a precise understanding of what a p-value means is extremely rare even among statistics lecturers, and, more interestingly, that there have always been fundamentally different understandings even when p-values were invented, e.g. between Fisher and Pearson.
Beyond that, there was a special issue recently in the american statistician discussing at length the issues of p-values in general and .05 in particular [2].
Personally, I of course feel that null hypothesis testing is often stupid and harmful, because null effects never exist in reality, effects without magnitude are useless, and because the "statelessness" of NHST creates or exacerbates problems such as publication bias and lack of power.
[1] http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
Consider effect sizes - this seems to be a completely different (yes important) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a massive effect size but if that is totally expected within your null model, then it is probably not all that exciting!
Effect sizes are a complement to some sort of hypothesis testing, but they are not a replacement.
> Prior probability.
yes, when you can effectively encode your prior probability I would say the posterior probability of seeing what you did is at least as good as p-value.
The thing that made me realize how ineffective P Also XCOM taught me that 98% != 100%
Xcom isn’t a good example because the game actively lies to you with displayed probabilities https://youtu.be/l0KEDYFWbVc
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
It's immensely frustrating that we haven't made a lot of progress since Cohen's (1962) paper.
Maybe tech industry insiders can tell me this ... but do real people actually make product decisions based solely on p < 0.05? Seems like the author is writing about a contrived problem.
Jokes aside, for product decisions (or all kinds of decisions, really) you should differentiate between statistical significance and relevance. A small measured difference in some metric, even if statistically significant to p < 0.000001, may not be relevant for a decision.
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…
Plus the interpretation is super straight forward: CI contains zero: Not significant. If nothing else, we should make CIs the primary default instead of p-values.
and shown an example of ,marketing using p values https://xkcd.com/882/
Earlier quoted context omitted.
Most big games implement "randomness" with "pseudorandomness" in the name of controlling variance of outcome, chopping off the long tail
Is there any game that does NOT use pseudorandom generators? And does this significantly change probabilities?
Doesnt change the overall probability, drastically reduces variance.
The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…
1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.)
2. P-values control type I error. Well-powered designs control type I and type II error. Good control of these errors is a kind of minimal requirement for a statistical procedure. Your example shows that we should perhaps consider more than just these aspects, but we should certainly be suspicious of any procedure that doesn't have good type I and II error control.
3. This is a problem with any kind of statistical modeling, and is not specific to p-values. All statistical techniques make assumptions that generally render them invalid when violated.