Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

81–90 of 168 posts

Re: Effect size is significantly more important than statistical significance

#81

Post by Tycho »

jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

What do you (or anyone else) think about the statistical conclusions in this paper? Particularly the adjusted r-squared values reported.

https://www.cambridge.org/core/journals/american-political-s...

Re: Effect size is significantly more important than statistical significance

#82

Post by candiodari »

DiabloD3 wrote: post #66
naasking wrote: post #40

Earlier quoted context omitted.

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.

You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.

Re: Effect size is significantly more important than statistical significance

#83

Post by robocat »

ummonk wrote: post #77

Agree with the title, but not the contents. The study in question is actually an example of a huge effect size (10% reduction in cases just from instructing villages they should wear masks is amazing) possibly hampered by poor statistical significance (as the blog post outlines).

Without knowing how many people were wearing masks, you can’t say the much about the 10% figure.

You get approximately[1] the same outcome if:

(a) masks are 100% effective but only 10% wear them, and

(b) masks are 10% effective and 100% wear them.

Is this study showing (a) or (b)?

Let us assume (b) masks only help by 10% and R0 is 2 without masks. If exponential transmission is occurring then in ~11.5 days you have the same number infected with masks as in 10 days without masks.

Either way the study has ended up with a 10% figure, and that figure gets misunderstood or intentionally misrepresented. If you want to argue for the effectiveness of masks against those that don’t wish to wear them, then personally I think it is a terrible study to argue with because 10% sounds shitty.

[1] Actual numbers depends on a heap of other things, but just assume those figures are right for the sake of making things easy to understand.

Disclaimer: I wear a mask during Level 2 lockdown in the South Island of New Zealand, and mask wearing has no partisan meaning here AFAIK.

Re: Effect size is significantly more important than statistical significance

#84

Post by sdenton4 »

Karrot_Kream wrote: post #73
Retric wrote: post #67

Earlier quoted context omitted.

Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc Here’s the experiment and here’s the data is concrete it may be bogus but it’s information. Updating probabilistic based on recursive estimates of probabilities is la…

> Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc This is called modelling error. Both Bayesian and frequentist approaches suffer from modelling error. That's what TFA talks about when mentioning the normality assu…

I wouldn't think of Black Swan events as tail events, so much as model failures or regime-changes. As in, 'we modeled this as a time-invariant gaussian distribution, but it's actually a mixture model where the second hidden mode was triggered in the aftermath of an asteroid strike that we didn't model for, because of course we didn't.'

In re, the arguey-person you were responding to, frequentist modeling is just as bad or worse for these sorts of situations.

Re: Effect size is significantly more important than statistical significance

#85

Post by rscho »

jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

If you were the king of science, I'd kindly ask you to think about replacing grant financing and all other financial incentives that go along with publishing. Now that would be efficient. 'Cause I currently make .05-barely-significant-results but if you force me to up my game I will provide .0001-barely-significant-results no problem, even with 'preregistration' or whatever hoop you hold in front of me.

As an aside, could you also please make medicine a real science, so I can finally scientifically demonstrate that my boss is wrong?

Re: Effect size is significantly more important than statistical significance

#86

Post by [deleted] »

Karrot_Kream wrote: post #73
Retric wrote: post #67

Earlier quoted context omitted.

Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc Here’s the experiment and here’s the data is concrete it may be bogus but it’s information. Updating probabilistic based on recursive estimates of probabilities is la…

> Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc This is called modelling error. Both Bayesian and frequentist approaches suffer from modelling error. That's what TFA talks about when mentioning the normality assu…

[deleted]

Re: Effect size is significantly more important than statistical significance

#88

Post by Retric »

sdenton4 wrote: post #84
Karrot_Kream wrote: post #73

Earlier quoted context omitted.

> Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc This is called modelling error. Both Bayesian and frequentist approaches suffer from modelling error. That's what TFA talks about when mentioning the normality assu…

I wouldn't think of Black Swan events as tail events, so much as model failures or regime-changes. As in, 'we modeled this as a time-invariant gaussian distribution, but it's actually a mixture model where the second hidden mode was triggered in the aftermath of an asteroid strike that we didn't model for, because of course we didn't.' In re, the arguey-person you were responding to, frequentist modeling is just as b…

Frequentist modeling isn’t useful, but that’s not how studies are evaluated. Let’s suppose your looking at a bunch of COVID studies and you ask yourself what if one or more of them was fraudulent?

Your investigation isn’t limited to the data provided by them it’s going to look for more information beyond the paper. This isn’t a failure of frequentist models because they evaluate the study and it’s output separately.

Re: Effect size is significantly more important than statistical significance

#89

Post by Retric »

Karrot_Kream wrote: post #73
Retric wrote: post #67

Earlier quoted context omitted.

Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc Here’s the experiment and here’s the data is concrete it may be bogus but it’s information. Updating probabilistic based on recursive estimates of probabilities is la…

> Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc This is called modelling error. Both Bayesian and frequentist approaches suffer from modelling error. That's what TFA talks about when mentioning the normality assu…

You don’t find black swans from the data you find them from building better models. You can look at 100 years of local flood and weather data to build up a flood assessment, but that’s not going to include mudslides or earthquakes etc. The same applies to studies.

My point is this: You can’t combine them using Bayesian statistics adjusting for the possibility of research fraud it’s simply not in the data.

Their great for well understood domains, less so for research. Frequentist models don’t work, but they also don’t even try.

PS: Math errors don’t really fall into modeling error.

Re: Effect size is significantly more important than statistical significance

#90

Post by derbOac »

robocat wrote: post #83
ummonk wrote: post #77

Agree with the title, but not the contents. The study in question is actually an example of a huge effect size (10% reduction in cases just from instructing villages they should wear masks is amazing) possibly hampered by poor statistical significance (as the blog post outlines).

Without knowing how many people were wearing masks, you can’t say the much about the 10% figure. You get approximately[1] the same outcome if: (a) masks are 100% effective but only 10% wear them, and (b) masks are 10% effective and 100% wear them. Is this study showing (a) or (b)? Let us assume (b) masks only help by 10% and R0 is 2 without masks. If exponential transmission is occurring then in ~11.5 days you have t…

Also, the study, IIRC, found greater social distancing in the mask conditions, which leads to other possible explanations.

I wear a mask all the time and am happy to but I agree this study, while solid in some respects, is not exactly overwhelming in making a compelling argument for masks.

Post reply on HN