P values are not as reliable as many scientists assume (2014) • Hacker News Board

Re: P values are not as reliable as many scientists assume (2014)

#81

Post by jsprogrammer » Wed, Sep 02, 2015, 9:09 PM UTC

kgwgk wrote: post #80

jsprogrammer wrote: post #78

Earlier quoted context omitted.

X can be any random variable that satisfies the requirements of the null-hypothesis. A more appropriate variable for your experiment would probably be the ratio of heads to tails (may need to add a bias to avoid division by 0). "you have a fair coin" is not a hypothesis, at least not a well-defined one.

Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. If you expect X to converge to C if the null hypothesis is true, you can simply say X->C. A proper notation involving probabilities woul…

>Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values.

Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typical null-hypothesis predicts).

To effectively use p-value analysis, it is typically assumed that your null-hypothesis predicts that your observations will be normally distributed with a mean of 0 and a standard deviation of 1. The total count of heads observed will not be distributed that way. Neither will the probability of a particular sequence (what your example seemed to be calculating). I say your null hypothesis is not well-defined because the term 'fair' remains undefined (though we could guess at the meaning) and in fact makes no predictions about the world. You need to apply transformations to your random variable so that it will appear normally distributed about 0 with a standard deviation of 1 if the hypothesis is true.

>Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1.

My notation is perfectly appropriate. X is a random variable and a random variable is the only thing that can go there (if you are doing p-value analysis). X is not assumed to be uniform or simple (although it certainly could be). P(T>T(X)|H) can be replaced with P(Y|H) every time (Y = T>T(X)).

>As I said, you're free to prefer your converging statistics and your well-defined null hypothesis. But you should be aware that people are talking about something completely different when discussing things like the 1e-7 p-value in the Higgs boson discovery or the reproducibility of statistically significant results.

I'm glad that we finally agree on this (although I dispute that anyone working on the Higgs boson discovery disagrees with me). One of my first claims was that others may not be calculating true p-values, but may calculate something and call it 'p-value' and then think that it means something it does not. In fact, this entire topic even links to an article in a prominent publisher claiming the same.

Do you think it is purely coincidental that the figures I showed you from the Higgs experiment show the lines converging towards only two different numbers: 1 and 0?

Edit: You'll have to give me some time on your edit. It's not something I typically calculate and I have other business to attend to today.

Re: P values are not as reliable as many scientists assume (2014)

#82

Post by kgwgk » Wed, Sep 02, 2015, 10:08 PM UTC

jsprogrammer wrote: post #81

kgwgk wrote: post #80

Earlier quoted context omitted.

Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Let me say that your notation is not very appropriate. It makes no sense to say that P(X|H) converges to 1. If you expect X to converge to C if the null hypothesis is true, you can simply say X->C. A proper notation involving probabilities woul…

>Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typic…

Ok, so maybe your definition does correspond to a p-value after all. It's hard to say as you have refused to discuss concrete cases (like the fair coin or the loaded die, which are standard examples to introduce p-values). But if you're actually calculating a p-value then it won't behave as you expect. It won't converge to anything (edit: if the null hypothesis holds). P-values are by definition uniformly distributed when the null hypothesis is true. If your "p-value" is not, then it's not a p-value. It really is that simple. Or maybe everyone else is using the wrong "p-values" and yours are the real thing. You can believe it if you want.

Please disregard my previous questions, I see no point in continuing this discussion. But you might want to read a bit more about p-values: you won't find anyone (I hope!) sharing your point of view. Once you understand what the p-value is, and what it is not, you might indeed conclude that they are entirely useless. Of course it's your right to avoid learning what p-values really are, and keep the faith. It's your choice.

Do you think it's purely coincidental that this this figure https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATL... includes the sigma=0 (null hypothesis) line at 0.5 and not at 1? (Hint: the expect value of the p-value under the null hypothesis is 0.5.) (That's a rhetorical question: I already know it's because this is not a well-formed experiment or something.)

Re: P values are not as reliable as many scientists assume (2014)

#83

Post by kgwgk » Wed, Sep 02, 2015, 11:55 PM UTC

marvy wrote: post #79

kgwgk wrote: post #61

Earlier quoted context omitted.

Assume that you're sampling from a normal distribution with known standard deviation sigma (1 for simplicity) and unknown mean mu. To test if the mean is larger than (the null hypothesis) mu=0 you can check if the observed value is larger than 1.64 sigma (for the 95% confidence test). So if your observation is larger than 1.64 you reject the null hypothesis. Your calculation would be correct only if the assumption "n…

I think I'm still missing something here. In particular, I'm still not getting 29% as a lower bound. I'm getting around 20%. If we compare mu=0 with mu=1.64, the probability density at x=1.64 is roughly 0.1 and 0.4, respectively, so the lower bound should be .1/(.1+.4)=1/5. No? Unless they were assuming something other than "two normal distributions with the same variance"?

You're absolutely right! My example was mainly for illustration, I was not sure that it would give exactly the same lower bound (but I was indeed surprised that it's below the 29% in those papers, which I thought a "hard" bound).

It seems the bound that you are calculating (that I have reproduced, R code below) was already published more than 50 years ago for this specific case of a normal distribution. See slide 10 in http://www.biostat.uzh.ch/teaching/master/previous/seminarba...

I have not really read the paper of Sellke et al. entirely, but it seems that the "calibration" they propose is more general but it makes some assumptions about the distribution of the p-value and it's therefore approximative.

  p=0.05
  null=0.5
  c0=qnorm(1-p)
  x=seq(0,5,0.01)
  y=100*dnorm(c0)/(dnorm(c0)+dnorm(x-c0))
  calib=100/(1-1/(exp(1)*p*log(p)))
  actual=min(y)
  plot(x,y,type="l",ylim=c(0,100),ylab="%null",xlab="mu1",bty="l",xaxs="i",yaxs="i")
  title(paste(null*100,"% null  p = ",p,sep=""))
  legend("topleft",c(paste("Sellke, Bayarri, Berger (2001) =",format(calib,digits=3)),
                   paste("Edwards, Lindman, Savage (1963) =",format(actual,digits=3))),
       lwd=2,col=c("red","blue"),bty="n")
  abline(h=calib,col="red")
  abline(h=actual,col="blue")
  grid()

Re: P values are not as reliable as many scientists assume (2014)

#84

Post by jsprogrammer » Thu, Sep 03, 2015, 12:44 AM UTC

kgwgk wrote: post #82

jsprogrammer wrote: post #81

Earlier quoted context omitted.

>Ok, so you're thinking about a random variable which converges to some value when the null hypothesis is true. This is fine, but it has nothing to do whatsoever with p-values. Well, the variable itself doesn't, just the observed value of P(X|H). X can be any random variable, but typically it will need to be transformed to have a normal distribution about 0 with a standard deviation of 1 (since this is what the typic…

Ok, so maybe your definition does correspond to a p-value after all. It's hard to say as you have refused to discuss concrete cases (like the fair coin or the loaded die, which are standard examples to introduce p-values). But if you're actually calculating a p-value then it won't behave as you expect. It won't converge to anything (edit: if the null hypothesis holds). P-values are by definition uniformly distributed…

>P-values are by definition uniformly distributed when the null hypothesis is true.

Where are you getting this from? When the null hypothesis is true, the p-value should be 1. This follows directly from the definition. If the p-value is not 1 and the null hypothesis is in fact true, your experiment or calculations are wrong. You might also just have the wrong null hypothesis (eg. sensors have more noise than assumed).

Re: P values are not as reliable as many scientists assume (2014)

#85

Post by marvy » Thu, Sep 03, 2015, 5:55 AM UTC

kgwgk wrote: post #83

marvy wrote: post #79

Earlier quoted context omitted.

I think I'm still missing something here. In particular, I'm still not getting 29% as a lower bound. I'm getting around 20%. If we compare mu=0 with mu=1.64, the probability density at x=1.64 is roughly 0.1 and 0.4, respectively, so the lower bound should be .1/(.1+.4)=1/5. No? Unless they were assuming something other than "two normal distributions with the same variance"?

You're absolutely right! My example was mainly for illustration, I was not sure that it would give exactly the same lower bound (but I was indeed surprised that it's below the 29% in those papers, which I thought a "hard" bound). It seems the bound that you are calculating (that I have reproduced, R code below) was already published more than 50 years ago for this specific case of a normal distribution. See slide 10…

I don't know R, but I found a few sites that happily run R code for me. I find the shape of that curve somehow pretty. 29% clearly can't be a hard bound, since we can get 4.8% by assuming no false negatives. I just wish I understood whether there is anything particularly natural about the number 29, or did they make their distributional assumptions for the same reasons you did: "mainly for illustration". If so, then the Nature article was terribly misleading by presenting that number as some kind of "speed of light"-type limit, because that makes p-values look worse than they really are. It seems that p-values are bad enough without making up more bad stuff about them! :)

Anyway, thanks for all your help. Your ability to dig up references (and pump out R code) at a moment's notice makes me think you are someone who knows quite a bit of statistics. I'll happily look at anything else you care to point me to.

Re: P values are not as reliable as many scientists assume (2014)

#86

Post by kgwgk » Thu, Sep 03, 2015, 6:55 AM UTC

marvy wrote: post #85

kgwgk wrote: post #83

Earlier quoted context omitted.

You're absolutely right! My example was mainly for illustration, I was not sure that it would give exactly the same lower bound (but I was indeed surprised that it's below the 29% in those papers, which I thought a "hard" bound). It seems the bound that you are calculating (that I have reproduced, R code below) was already published more than 50 years ago for this specific case of a normal distribution. See slide 10…

I don't know R, but I found a few sites that happily run R code for me. I find the shape of that curve somehow pretty. 29% clearly can't be a hard bound, since we can get 4.8% by assuming no false negatives. I just wish I understood whether there is anything particularly natural about the number 29, or did they make their distributional assumptions for the same reasons you did: "mainly for illustration". If so, then…

Assuming no false negatives is just not an option :-) I think that can only happen if the situation is such that there can be no false positives either (i.e. the p-value when the null hypothesis is not true is always zero). EDIT: What I wrote is true only if the distributions under the null and the alternative are completely disjoint. You can actually have very low false negative rates if the distributions are not symmetric, and if you allow different distributions you can do even better: imagine the null hypothesis is x~Normal(0,1) and the alternative is x=C0=1.64 (exactly the cutoff value for 0.05 significance). If we get exactly p=0.05 then the probability of the null being true is 0%. I mean, we get p in [C0-epsilon C0+epsilon] with probability 1 under the alternative, but with probability->0 under the null as epsilon->0. Of course, this alternative is very unlikely and mixing continous and discrete distributions is always tricky. This is why it makes sense to make averages over prior distributions of the alternative.

As you can see in slide 14, there are multiple calibrations proposed under different assumptions. I agree it is misleading to give one as the "real" error I rate, but it's interesting that all of them are giving rates well above the nominal alpha rate. EDIT: note as well that this is for the case where in 50% of the cases the null hypothesis is true(nowhere in the calculation of p-values do we consider how often the null hypothesis is true, but obviously if it's always true 100% of the significant results will be false positives and if it's never true 0% of the significant results will be false positives).

In slide 11 there are other calculations for the normal case, two-sided test this time. But instead of looking for the mu1 giving the lowest bound, they calculate the aggregate error rate making some assumptions about the distribution of the mu1. For example, if I understand correctly the results, assuming mu1 is normally distributed around mu0=0, if you get a p-value=0.05 (in the two-sided test, some modifications are required to the calculation we did) you should expect the null hypothesis to be true at least 32.1% (if the distribution of mu1 is very concentrated around 0, the 50% rejection rate on the left side of the chart dominates, if the standard deviation if very high the region of almost 100% rejection rate far from mu0 at the right of the chart dominates, for some intermediate standard deviation one will hopefully get the 32.1% lower bound).

Unfortunately, I think the assumption behind the nice result 1/(1-1/(e p log(p)))) is that p-values follow a beta distribution when the null hypothesis is not true and I don't think there is a clear interpretation of that.

Re: P values are not as reliable as many scientists assume (2014)

#87

Post by marvy » Thu, Sep 03, 2015, 8:57 AM UTC

kgwgk wrote: post #86

marvy wrote: post #85

Earlier quoted context omitted.

I don't know R, but I found a few sites that happily run R code for me. I find the shape of that curve somehow pretty. 29% clearly can't be a hard bound, since we can get 4.8% by assuming no false negatives. I just wish I understood whether there is anything particularly natural about the number 29, or did they make their distributional assumptions for the same reasons you did: "mainly for illustration". If so, then…

Assuming no false negatives is just not an option :-) I think that can only happen if the situation is such that there can be no false positives either (i.e. the p-value when the null hypothesis is not true is always zero). EDIT: What I wrote is true only if the distributions under the null and the alternative are completely disjoint. You can actually have very low false negative rates if the distributions are not sy…

Ok, so the pretty result is mostly arbitrary. Fair enough. Re: false negatives... you seem to be living in a world of bell curves, or at least a mostly continuous world. I can easily make (very contrived) experiments where false negatives just don't happen. For instance: I have two coins. One is a perfectly fair coin. The other is a two-headed coin. You see me flip one of them. The null hypothesis is that I flipped the fair coin. A false negative means deciding the coin is fair but it's really not. This will never happen, because you will only decide that if it lands tails, and then it must be fair. (If I only flip it once, the false positive rate is something like 1/3, not 0.) But this is probably much too contrived for your taste, and maybe even for mine. But it's almost 5am, and I must go to sleep now, or else it will get bright soon, and I never will. I now appreciate the value of the 20min procrastination setting.

Re: P values are not as reliable as many scientists assume (2014)

#88

Post by kgwgk » Thu, Sep 03, 2015, 11:43 AM UTC

marvy wrote: post #87

kgwgk wrote: post #86

Earlier quoted context omitted.

Assuming no false negatives is just not an option :-) I think that can only happen if the situation is such that there can be no false positives either (i.e. the p-value when the null hypothesis is not true is always zero). EDIT: What I wrote is true only if the distributions under the null and the alternative are completely disjoint. You can actually have very low false negative rates if the distributions are not sy…

Ok, so the pretty result is mostly arbitrary. Fair enough. Re: false negatives... you seem to be living in a world of bell curves, or at least a mostly continuous world. I can easily make (very contrived) experiments where false negatives just don't happen. For instance: I have two coins. One is a perfectly fair coin. The other is a two-headed coin. You see me flip one of them. The null hypothesis is that I flipped t…

I agree on your point, if we are sufficiently creative we can get many extreme results. For example, I made an addition to the first paragraph of my previous comment, that you might have missed, giving an example where the probability of the null hypothesis being true when p=0.05 is zero (or arbitrarily small, if we replace the discrete probability lump under the alternative hypothesis by a continuous distribution which is concentrated enough). I also added a comment on the second paragraph, by the way.

One minor comment on your example. If H0:fair coin and H1:two-headed and the statistic is the number of heads h, I cannot reject (at the 0.05 level) the null when n (the number of flips) is small even if I'm only getting heads. For one flip, p[h=1|H0]=0.5. For two flips p[h=2|H0]=0.25. For n>5 you will of course reject the null hypothesis for every case where H1 is true (and for ~5% of the cases where H0 is true). There will be no false negatives. But I guess you have noticed that this doesn't help with the false discovery rate in this example: when H1 is true the p-value will be very small (1/2^n) so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails).

Ok, enough time wasted on this subject :-)

Re: P values are not as reliable as many scientists assume (2014)

#89

Post by marvy » Thu, Sep 03, 2015, 7:13 PM UTC

kgwgk wrote: post #88

marvy wrote: post #87

Earlier quoted context omitted.

Ok, so the pretty result is mostly arbitrary. Fair enough. Re: false negatives... you seem to be living in a world of bell curves, or at least a mostly continuous world. I can easily make (very contrived) experiments where false negatives just don't happen. For instance: I have two coins. One is a perfectly fair coin. The other is a two-headed coin. You see me flip one of them. The null hypothesis is that I flipped t…

I agree on your point, if we are sufficiently creative we can get many extreme results. For example, I made an addition to the first paragraph of my previous comment, that you might have missed, giving an example where the probability of the null hypothesis being true when p=0.05 is zero (or arbitrarily small, if we replace the discrete probability lump under the alternative hypothesis by a continuous distribution wh…

> so if the observed p-value is ~0.05 (or any other value larger than 1/2^n) then it's for sure a false positive (because there will be at least one occurrence of tails).

Good point! I didn't think of that.

> Ok, enough time wasted on this subject :-)

Even better point!

Re: P values are not as reliable as many scientists assume (2014)

#90

Post by jsprogrammer » Fri, Sep 04, 2015, 4:07 PM UTC

jsprogrammer wrote: post #84

kgwgk wrote: post #82

Earlier quoted context omitted.

Ok, so maybe your definition does correspond to a p-value after all. It's hard to say as you have refused to discuss concrete cases (like the fair coin or the loaded die, which are standard examples to introduce p-values). But if you're actually calculating a p-value then it won't behave as you expect. It won't converge to anything (edit: if the null hypothesis holds). P-values are by definition uniformly distributed…

>P-values are by definition uniformly distributed when the null hypothesis is true. Where are you getting this from? When the null hypothesis is true, the p-value should be 1. This follows directly from the definition. If the p-value is not 1 and the null hypothesis is in fact true, your experiment or calculations are wrong. You might also just have the wrong null hypothesis (eg. sensors have more noise than assumed)…

I'm willing to accept that in some designs, the first observation of p-value may be uniformly distributed over (0,1], but, as additional observations are made, the value should converge to 0 or 1. What would be the purpose, or usefulness of p-value being uniformly distributed if the null-hypothesis is true? It's much simpler to design things to converge to a single number.

Edit: I have also considered that p-value could be uniformly distributed if the null-hypothesis is false (where you claimed true). I don't know the answer to that.