Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

61–70 of 168 posts

Re: Effect size is significantly more important than statistical significance

#61

Post by bongoman37 »

phreeza wrote: post #6
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

This is clearly a cost/benefit tradeoff, and the sweet spot will depend entirely on the field. If you are studying the behavior of heads of state, getting an additional N is extremely costly, and having a p=0.05 study is maybe more valuable than having no published study at all, because the stakes are very high and even a 1% chance of (for example) preventing nuclear war is worth a lot. On the other hand, if you are…

What if it's the other way round and a p<0.05 study says that the best way to make sure a rival country does not do a nuclear strike on you first is to do a massive nuclear strike on them first?

Re: Effect size is significantly more important than statistical significance

#62

Post by exporectomy »

vkou wrote: post #32
robbedpeter wrote: post #21

Earlier quoted context omitted.

The USDA food pyramid and nutrition education would suggest that there's an inherent danger in just letting people believe irrational things after a correction is known. It depends on the belief - flat earth people aren't likely to cause any harm. Bad nutrition information can wreak havoc at scale.

Flat earth beliefs doesn't cause harm, but flat earth believers have largely upgraded to believing more dangerous nonsense.

Data or it didn't happen. This really sounds like you're inventing a caricature of your enemy and assigning them "dangerous" qualities so you can hate them more.

Re: Effect size is significantly more important than statistical significance

#63

Post by Retric »

tux3 wrote: post #49
Retric wrote: post #38

Earlier quoted context omitted.

Bayesian reasoning has even worse underpinnings. You don’t actually know any of the things the equations want. For example suppose a robot is counting Red and Blue balls from a bin, the count is 400Red and 637Blue, it just classified a Red ball. Now what’s the count, wait what’s the likelihood it misclassified a ball? How accurate are those estimates, and those estimates of those ... For a real world example someone…

Suppose the likelihood it missclassified a ball is significantly different from zero, but not yet known precisely. If you use a model that doesn't ask you to think about this likelihood at all, you will get the same result as if you had used bayes and consciously chose to approximate the likelihood of misclassification as zero. You may get slightly better results if you have a reasonnable estimate of that probability…

The criticism is important because of how Bayes keeps using the probability between experiments. Garbage in Garbage out.

As much as people complain about frequentist approaches, examining the experiment independently from the output of the experiment effectively limits contamination.

Re: Effect size is significantly more important than statistical significance

#64

Post by exporectomy »

bee_rider wrote: post #31
exporectomy wrote: post #14

Earlier quoted context omitted.

Only a problem for people who are trying hard not to think. You can just ignore those people. They're not doing any harm believing their beliefs.

We are literally in the middle of a global crisis that is founded on people misunderstanding science.

What on earth are you talking about? I guess climate change but that's certainly not founded on people misunderstanding science, it's caused by people understanding science which led to industrialization. Or maybe you mean covid-19? Neither that. You're just trying to make it seem like it's somehow very serious and bad if everyone doesn't agree with you. It's not.

Re: Effect size is significantly more important than statistical significance

#65

Post by sampo »

lotu wrote: post #24
sanxiyn wrote: post #19

Mask's effect size on seroprevalence is probably zero. So no effect is expected result. That's because mask acts on R0, not seroprevalence. After acting on R0, if R0 is >1, exponential growth, if 1 to <1.

Also, they aren't testing masking effect on seroprevalence (or R0), they are testing the effect of sending out free masks and encouraging masking. That is only going to move the percent of people masking up or down a few percent at best.

The study says:

> The intervention increased proper mask-wearing from 13.3% in control villages (N=806,547 observations) to 42.3% in treatment villages (N=797,715 observations)

https://www.poverty-action.org/sites/default/files/publicati...

Re: Effect size is significantly more important than statistical significance

#66

Post by DiabloD3 »

naasking wrote: post #40
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.

Re: Effect size is significantly more important than statistical significance

#67

Post by Retric »

tfehring wrote: post #44
Karrot_Kream wrote: post #41

Earlier quoted context omitted.

Huh? You can derive all of those from Bayesian models. If you're counting balls from a bin with replacement, and your bot has counted 400Red with 637Blue, you have a Beta/Binomial model. That means you p_blue | data ~ Beta(401, 638) assuming a Uniform prior. The probability of observing a red ball given the above p_blue | data is P(red_obs | p_blue) = 1 - P(blue_obs | p_blue), which is calculable from p_blue | data.…

And if misclassification is a concern (as the parent mentioned) you can put a prior on that rate too!

Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc

Here’s the experiment and here’s the data is concrete it may be bogus but it’s information. Updating probabilistic based on recursive estimates of probabilities is largely restating your assumptions. Black swans can really throw a wrench into things.

Plenty of downvotes and comments, but nothing addressing the point of the argument might suggest something.

Re: Effect size is significantly more important than statistical significance

#68

Post by vkou »

exporectomy wrote: post #62
vkou wrote: post #32

Earlier quoted context omitted.

Flat earth beliefs doesn't cause harm, but flat earth believers have largely upgraded to believing more dangerous nonsense.

Data or it didn't happen. This really sounds like you're inventing a caricature of your enemy and assigning them "dangerous" qualities so you can hate them more.

Nobody needs to caricature the insane beliefs surrounding COVID (or flat earth), people holding them are doing a good enough job of that themselves.

I do have a few favorites. "COVID tests give you COVID, so I won't go get tested" is certainly up there. I can't say I give two figs about your opinion on the Earth's topology, but this one is a public health problem, that's crippling hospitals around the country.

Re: Effect size is significantly more important than statistical significance

#69

Post by romwell »

DiabloD3 wrote: post #66
naasking wrote: post #40

Earlier quoted context omitted.

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.

Also replication studies for negative or null results in addition to positive ones (we don't have either).

Re: Effect size is significantly more important than statistical significance

#70

Post by rootusrootus »

nix0n wrote: post #34
tommiegannert wrote: post #5

Earlier quoted context omitted.

A few years ago, HN comments complained about the censorship that only leaves successful studies. We need to report on everything we've tried, so we don't walk around on donuts. What's missing in my mind is admitting that results were negative. I'm reading up on financial literacy, and many studies end with some metrics being "great" at p 5%, but then some other metrics are also "great" at p 10%, without the author e…

> ...so we don't walk around on donuts I agree with what you're saying, but I don't understand this phrase.

I don't know where that turn of phrase comes from, but I imagine it's synonymous with 'walking around in circles'.
Post reply on HN