Earlier quoted context omitted.
Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut. Were I to run another 100 independently designed experiments that all teste…
> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unb…
That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived.
You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.