Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not just p=0.048 vs. p=0.052

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

81–90 of 92 posts

Re: It’s not just p=0.048 vs. p=0.052

#81

Post by kgwgk »

mehrdadn wrote: post #29
firebacon wrote: post #25

Earlier quoted context omitted.

Check out these links [0] [1] or google for "p value distribution" or "p curve" [0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t... [1] https://en.wikipedia.org/wiki/P-value#Distribution

I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely. Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused…

> How is p uniformly distributed under H0?

That's the very definition of a p-value! The mapping of data to p-values is chosen to have a uniform distribution of p-values when the data is distributed according to the null hypothesis. That's the property that makes p-values interesting.

Re: It’s not just p=0.048 vs. p=0.052

#82

Post by kgwgk »

blauditore wrote: post #60

> Also, to get technical for a moment, the p-value is not the “probability of happening by chance.” Is it not? According to Wikipedia, it's "[...] the probability that, when the null hypothesis is true, the statistical summary [...] would be equal to, or more extreme than, the actual observed results." This sounds pretty much like "probability of happening by chance".

It is not the “probability of [happening by chance]” (as opposed to the probability of [happening for some reason]).

It is the “[probability of happening] by chance” (as opposed to the [probability of not happening] by chance).

Re: It’s not just p=0.048 vs. p=0.052

#83

Post by nerdponx »

kgwgk wrote: post #77
mehrdadn wrote: post #20

Earlier quoted context omitted.

I think that's what he's saying too, but what is that supposed to show? Is he arguing against some claim that every interval of 1 standard deviation is equally significant? Did anybody make this claim? So far as I know, nobody considers (say) a 6-sigma effect to be 6 times stronger than a 1-sigma effect...

I don’t get it either. It’s a trivial consequence of having a threshold: if we say two cities are “far” when they are at least 1000 miles away then Washington D.C. is not far from Jacksonville while Boston is far from Jacksonville, even though Boston is not far from Washington.

That's not quite the problem.

Let's assume you've already decided in advance what "far" means.

Without moving either city from its current location, the same experiment can give you "very far" and "very close" in identical replications.

Re: It’s not just p=0.048 vs. p=0.052

#84

Post by mehrdadn »

dash2 wrote: post #37
mehrdadn wrote: post #29

Earlier quoted context omitted.

I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely. Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused…

They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution! Here's…

So I don't have (or know) R, but I do have access to Mathematica, and this is most definitely not giving me a uniform distribution (how could it?!):
Why is your x variable though? If H0 is true shouldn't your x be fixed?

I think if you remove the subtraction though then you do get a uniform distribution -- in which case I see what the claim is, yeah. Wasn't really clear to me earlier but indeed, getting p = 5% means you have a 5% chance of getting observations that extreme, so I guess it is uniformly distributed!

Re: It’s not just p=0.048 vs. p=0.052

#85

Post by SilasX »

swsieber wrote: post #48
rom1v wrote: post #33

Related, about p-values: > Here's the problem in a nutshell: If you run 1000 experiments over the course of your career, and you get a significant effect (p > […] However, this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, a…

For about a year or so now Ive been wanting to make a game about science. It'd basically be a research and discovery simulator, and there would be a free-play mode. Some of the knobs would be # of required replication, required p-value and how good people are at generating hypothesis. I think it'd be eye opening.

Yes! Riffing off that, I'd want to do a version where all the data in the game is pure randomness, but where your experimental configurations can influence the outcome. (e.g. you can clean the equipment, which alters the reading, or redo experiments that "went wrong")

You'd be given a prize/goal for publishable findings. Then you'd gradually introduce enough bias into the experiments to get something publishable, and then get hit with the reveal that "oh you were generating effects from random data, jerk".

(Okay, maybe that's overcomplicating it.)

Re: It’s not just p=0.048 vs. p=0.052

#86

Post by kgwgk »

nerdponx wrote: post #83
kgwgk wrote: post #77

Earlier quoted context omitted.

I don’t get it either. It’s a trivial consequence of having a threshold: if we say two cities are “far” when they are at least 1000 miles away then Washington D.C. is not far from Jacksonville while Boston is far from Jacksonville, even though Boston is not far from Washington.

That's not quite the problem. Let's assume you've already decided in advance what "far" means. Without moving either city from its current location, the same experiment can give you "very far" and "very close" in identical replications.

I’m commenting on pfortuny’s (correct) interpretation “those events are only separated be 1.1std deviations, which is little” of the original post by Gelman (the difference between a significant result and a non-significant result may not be significant = a city which is far from Jacksonville and a city which is not far from Jacksonville may not be far from each other).

Re: It’s not just p=0.048 vs. p=0.052

#87

Post by Quekid5 »

thanatropism wrote: post #32
tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

P-values are a sub-optimal but okay-ish of quantifying a Popperian hypothesis (a designed-to-be-refutable conjecture). The mathematics is not the problem, the problem is carving science (which in my view (and Quine's and others's) is pretty much defined by the unity of science) in testable morcels. None of the great achievements of science (Newton, Darwin, Mendeleyev, etc.) were obtained on the basis of Popperian dem…

Turns out that once you invent a system for "acceptance" (p < 0.05 or whatever), people will game it. Who knew?

Re: It’s not just p=0.048 vs. p=0.052

#88

Post by tgb »

BenoitEssiambre wrote: post #41
tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

That interpretation makes the test basically useless because we know a priori that any two variable for things within each others light cone affect each other at least a little. More practically, since the test tells you nothing about size of effect, it will pick up on the tiniest of bias in your experimental procedure and always reject the null if you have enough data. From the author of the article: "The general po…

Compound/interval null hypotheses basically solve this effect-size problem and probably should be used more.

Re: It’s not just p=0.048 vs. p=0.052

#89

Post by BenoitEssiambre »

tgb wrote: post #88
BenoitEssiambre wrote: post #41

Earlier quoted context omitted.

That interpretation makes the test basically useless because we know a priori that any two variable for things within each others light cone affect each other at least a little. More practically, since the test tells you nothing about size of effect, it will pick up on the tiniest of bias in your experimental procedure and always reject the null if you have enough data. From the author of the article: "The general po…

Compound/interval null hypotheses basically solve this effect-size problem and probably should be used more.

Yes, it should be required.
Post reply on HN