Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

21–30 of 168 posts

Re: Effect size is significantly more important than statistical significance

#21

Post by robbedpeter »

exporectomy wrote: post #14
modeless wrote: post #8

Earlier quoted context omitted.

Not only is it not valuable to publish tons of studies with p=.04999 and small effect size, in fact it's harmful. With so many questionable results published in supposedly reputable places it becomes possible to "prove" all sorts of crackpot theories by selectively citing real research. And if you try to dispute the studies you can get accused of being anti-science.

Only a problem for people who are trying hard not to think. You can just ignore those people. They're not doing any harm believing their beliefs.

The USDA food pyramid and nutrition education would suggest that there's an inherent danger in just letting people believe irrational things after a correction is known. It depends on the belief - flat earth people aren't likely to cause any harm. Bad nutrition information can wreak havoc at scale.

Re: Effect size is significantly more important than statistical significance

#22

Post by robertlagrant »

civilized wrote: post #15
exporectomy wrote: post #9

I wonder if we should separate the roles of scientist and researcher. Universities would have generalist "scientists" who's job would be to consult for domain-specialized researchers to ensure they're doing the science and statistics correctly. That way, we don't need every researcher in every field to have a deep understanding of statistics, which they often don't. Either that or stop rewarding such bad behavior. Sc…

The scientific establishment will never be convinced to stop doing bad statistics, so "the solution to bad speech is more speech". Statisticians should be rewarded for effective review and criticism of flawed studies, and critical statistical reviews of any article should be easy to find when they exist. This is sounding like a great startup idea for a new scientific journal, actually.

Just adding an Arxiv filter that allows me to set a minimum p-value or variation % would do it!

Re: Effect size is significantly more important than statistical significance

#23

Post by vavooom »

civilized wrote: post #15
exporectomy wrote: post #9

I wonder if we should separate the roles of scientist and researcher. Universities would have generalist "scientists" who's job would be to consult for domain-specialized researchers to ensure they're doing the science and statistics correctly. That way, we don't need every researcher in every field to have a deep understanding of statistics, which they often don't. Either that or stop rewarding such bad behavior. Sc…

The scientific establishment will never be convinced to stop doing bad statistics, so "the solution to bad speech is more speech". Statisticians should be rewarded for effective review and criticism of flawed studies, and critical statistical reviews of any article should be easy to find when they exist. This is sounding like a great startup idea for a new scientific journal, actually.

I do enjoy the idea of a journal focused entirely on the review of statistical methods and underlying methodologies applied in modern day research. Could act as a helpful signal for relevant and applicable research.

Re: Effect size is significantly more important than statistical significance

#24

Post by lotu »

sanxiyn wrote: post #19

Mask's effect size on seroprevalence is probably zero. So no effect is expected result. That's because mask acts on R0, not seroprevalence. After acting on R0, if R0 is >1, exponential growth, if 1 to <1.

Also, they aren't testing masking effect on seroprevalence (or R0), they are testing the effect of sending out free masks and encouraging masking. That is only going to move the percent of people masking up or down a few percent at best.

Re: Effect size is significantly more important than statistical significance

#25

Post by sanxiyn »

mrtranscendence wrote: post #20

> If most effect sizes are small or zero, then most interventions are useless. But this doesn't necessarily follow, does it? If there really were a 1.1-fold reduction in risk due to mask-wearing it could still be beneficial to encourage it. The salient issue (taking up most of the piece) seems to be not the size of the effect but rather the statistical methodology the authors employed to measure that size. The p-valu…

I agree the problem here is an incorrect model. Mask does not act on seroprevalence. Measuring mask's effect on seroprevalence is just wrong study design, although it may be easier to do.

Re: Effect size is significantly more important than statistical significance

#26

Post by twoslide »

Robotbeat wrote: post #7
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it…

> when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that.

That's not necessarily true in social sciences. When you're working with large survey datasets, many variables are significantly related. That doesn't mean these relationships are meaningful or causal, they could be due to underlying common causes, etc. (Maybe social sciences weren't included in "real science" - but there's where a lot of stats discussions focus)

Re: Effect size is significantly more important than statistical significance

#27

Post by mumblemumble »

Robotbeat wrote: post #7
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it…

Fine. Do it like the experimental physicists do: if you think you're on to something, refine and repeat the experiment in order to get a more robust, repeatable result.

The original sin of the medical and social sciences is failing to recognize a distinction between exploratory research and confirmatory research and behave accordingly.

Re: Effect size is significantly more important than statistical significance

#29

Post by [deleted] »

mumblemumble wrote: post #27
Robotbeat wrote: post #7

Earlier quoted context omitted.

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it…

Fine. Do it like the experimental physicists do: if you think you're on to something, refine and repeat the experiment in order to get a more robust, repeatable result. The original sin of the medical and social sciences is failing to recognize a distinction between exploratory research and confirmatory research and behave accordingly.

[deleted]

Re: Effect size is significantly more important than statistical significance

#30

Post by robocat »

From the article:

Ernest Rutherford is famously quoted proclaiming “If your experiment needs statistics, you ought to have done a better experiment.”

“Of course, there is an existential problem arguing for large effect sizes. If most effect sizes are small or zero, then most interventions are useless. And this forces us scientists to confront our cosmic impotence, which remains a humbling and frustrating experience.”

Post reply on HN