Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

151–160 of 168 posts

Re: Effect size is significantly more important than statistical significance

#151

Post by davidmanheim »

jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

Maybe all studies should be preregistered, including their methods... like this one was?

https://osf.io/vzdh6/

Re: Effect size is significantly more important than statistical significance

#152

Post by davidmanheim »

davidmanheim wrote: post #131
naasking wrote: post #40

Earlier quoted context omitted.

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

This paper was pretty clearly pre-specified here; https://files.givewell.org/files/DWDA%202009/IPA/Masks_RCT_P...

And it was actually preregistered as well: https://osf.io/vzdh6/

Re: Effect size is significantly more important than statistical significance

#153

Post by BoiledCabbage »

MaulingMonkey wrote: post #51
Robotbeat wrote: post #7

Earlier quoted context omitted.

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it…

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Mi…

That's a nice post and all, but none of that had anything to do with reliability. In all of those cases, those components worked exactly as designed when operating within their specification ranges (ie inserters insert when they have power).

The point is, it would be significantly more complex if things frequently failed even when "operating properly". And this happened at all levels of abstraction in a factory.

Re: Effect size is significantly more important than statistical significance

#154

Post by BeFlatXIII »

shakezula wrote: post #3
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

I blame most of this on pop science. It's absolutely ruined the average public's respect for the behind the scenes work doing interesting stuff in every field. What's worse is the attitude it breeds. Anti-intellectualism runs rampant amongst even well educated members of my social circle. It's frustrating to say the least.

"Believe the science" vs. "understand the process". The former merely uses the language of science to gain legitimacy.

Re: Effect size is significantly more important than statistical significance

#155

Post by setgree »

LeonB wrote: post #133
setgree wrote: post #121

Earlier quoted context omitted.

> or a p value of something like p = 0.0001 This has been proposed [0], albeit for a threshold of p Here's Andy Gelman and others arguing otherwise [1]. They also got like 800 scientists to sign on to the general idea of no longer using statistical significance at all [2]. [0] https://www.nature.com/articles/s41562-017-0189-z [1] http://www.stat.columbia.edu/~gelman/research/unpublished/ab... [2] https://www.nature.c…

Given the (estimated) number of scientists in the world and their general propensity to sign on to something… is 800 scientists a significant amount?

I don't know, and it's a fair point; I think I should have just summarised the follow-up article as "here's some follow-up by the authors for context"

Re: Effect size is significantly more important than statistical significance

#156

Post by vannevar »

ummonk wrote: post #77

Agree with the title, but not the contents. The study in question is actually an example of a huge effect size (10% reduction in cases just from instructing villages they should wear masks is amazing) possibly hampered by poor statistical significance (as the blog post outlines).

It should also be noted that the positive effect rose to 35% for people over the age of 60, who make up the overwhelming majority of serious Covid-19 cases. The omission of this important fact from the article leads me to question the motivation of the author.

Re: Effect size is significantly more important than statistical significance

#157

Post by hprotagonist »

harry8 wrote: post #134
hprotagonist wrote: post #108

Earlier quoted context omitted.

we tried using 0.1 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.12 mL, it didn't work we tried using 0.13 mL, it didn't work

we tried using 0.10 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.13 mL, it didn't work we tried using 0.15 mL, it didn't work we tried using 0.17 mL, it didn't work we tried using 0.16 mL, it didn't work we tried using 0.18 mL, it didn't work we tried using 0.20 mL, it didn't work we tried using 0.14 mL, it didn't work we tried using 0.12 mL, it worked so we published Do you want to know…

i don’t want to know about each test that didn’t work as a separate publication, that’s for sure!

Re: Effect size is significantly more important than statistical significance

#158

Post by ramblenode »

bluGill wrote: post #4
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

> Plus, the idea that we can remove such small, noisy confounding factors is just silly. We need to look for the things that stand out from that noise floor We have found most of them, and all the easy ones. Today the interesting things are near the noise floor. 3000 years ago atoms were well below the noise floor, now we know a lot about them - most of it seems useless in daily life yet a large part of the things we…

> We have found most of them, and all the easy ones. Today the interesting things are near the noise floor.

The noise floor is not static. A major theoretical advance spurs an advance in instrumentation, which then supports more science. The hypothesis space is usually much larger than the data space, making the bottleneck theory, not data. The "end of progress" has been lamented again and again since before Galileo, only to be upended by a paradigm shifting theory that paved the way for lots of new science. Many of these theories were developed long after the data and instruments were available, and were produced with relatively simple data: Young's double slit experiment, Mendelian genetics, the photoelectric effect, Brownian motion, most of classical mechanics, quantum teleportation, BOLD MRI, etc.

Re: Effect size is significantly more important than statistical significance

#159

Post by carlmr »

hprotagonist wrote: post #157
harry8 wrote: post #134

Earlier quoted context omitted.

we tried using 0.10 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.13 mL, it didn't work we tried using 0.15 mL, it didn't work we tried using 0.17 mL, it didn't work we tried using 0.16 mL, it didn't work we tried using 0.18 mL, it didn't work we tried using 0.20 mL, it didn't work we tried using 0.14 mL, it didn't work we tried using 0.12 mL, it worked so we published Do you want to know…

i don’t want to know about each test that didn’t work as a separate publication, that’s for sure!

That's true, we would need a way to collect this data so it's easily seen as part of a whole.

E.g. if you search for eggs and cholesterol you should find all studies with their summarized results on whether eggs are ok or not for your cholesterol, grouped by researcher, so if somebody does 200 studies to find the one positive it's instantly visible.

Re: Effect size is significantly more important than statistical significance

#160

Post by shakezula »

shawn-butler wrote: post #58
shakezula wrote: post #3

Earlier quoted context omitted.

I blame most of this on pop science. It's absolutely ruined the average public's respect for the behind the scenes work doing interesting stuff in every field. What's worse is the attitude it breeds. Anti-intellectualism runs rampant amongst even well educated members of my social circle. It's frustrating to say the least.

Some say that it is not anti-intellectualism to realize the emperor has no clothes but enlightenment. Either way it’s dangerous.

It can be both, but you're absolutely right.
Post reply on HN