Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

141–150 of 168 posts

Re: Effect size is significantly more important than statistical significance

#141

Post by TameAntelope »

q-big wrote: post #94
bluGill wrote: post #78

Earlier quoted context omitted.

You missed the working part. Success was a prerequisite to their after the fact feelings. At least some of the control group will be in old age but still alive when we know it woris. They might not know if it is infinite life (and side effects may turn it into die at 85, so some control may outlive the intervention group after the study ), but they will know on average they did worse

> At least some of the control group will be in old age but still alive when we know it woris. The anti-aging serum could work (i.e. make you older), but have strong negative side effects.

So then it wouldn't "work", where "work" as used here is defined, "to function or operate according to plan or design".

And no, it's not reasonable to assume I meant "work" as in, "have an anti-aging serum that has strong negative side effects."

Re: Effect size is significantly more important than statistical significance

#142

Post by funklute »

lmm wrote: post #140
funklute wrote: post #139

Earlier quoted context omitted.

> At which point you've just found a more cumbersome way to do frequentist statistics. Hmm, in one way, yes...but on the other hand, Bayesian posteriors are a lot more intuitive to interpret, for most people. So I think you trade one form of convenience for another. But as you sort of hint at, the results should usually be fairly similar, whether you're doing frequentist or Bayesian analysis. So in most cases, I doub…

> So in most cases, I doubt it matters that much. Where it does matter, is when you have grounds for strong priors, that you want to take advantage of. In such cases you can improve your chances of being correct in the "here and now", if you do a Bayesian analysis. I completely agree with this - but it's exactly this dynamic that I think, at least in the current academic environment, does more harm than good. Effecti…

> letting them pick their own prior multiplies that kind of thing many times over.

I'm a big fan of sensitivity analysis in this context. Don't just pick one prior and call it a day, but show the effect of having liberal vs conservative priors, and discuss that in light of the domain knowledge. That gives the next researcher a much better foundation than a single prior, or a p-value, ever could.

Unfortunately, if it was a non-trivial paper to begin with, it now just turned into a whole book.

Re: Effect size is significantly more important than statistical significance

#143

Post by candiodari »

captain_price7 wrote: post #111
candiodari wrote: post #82

Earlier quoted context omitted.

You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.

Why would someone want to game a negative result? Nobody ever becomes famous for saying my approach doesn't work. (As long as science is open, to make sure there is actually good work done by researchers before reaching this neg result.)

To have their name on a publication, which is a currency in the academic world.

Re: Effect size is significantly more important than statistical significance

#144

Post by dan_mctree »

hyperbovine wrote: post #17
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

Come into Bayesian land, the water is fine. The whole NHST edifice starts to seem really shaky once you stop and wonder if "True" and "False" are really the only two possible states of a scientific hypothesis. Andrew Gelman has written about this in many places, e.g. http://www.stat.columbia.edu/~gelman/research/published/aban... .

It's interesting that high p-values actually seem to more conclusively state something than low p values (like p With a high p value, you can say with some degree of certainty that your test was unable to detect any effect. Whether it was due to the lack of an effect or because your test wasn't capable of measuring it

With a low p value, you don't actually really know if you detected something interesting. It could be due to a flawed test, biases, non-causal correlations, faulty p-hacky stats, etc.

So why do we consider the latter more worthwhile when it seems to say less?

Re: Effect size is significantly more important than statistical significance

#145

Post by q-big »

TameAntelope wrote: post #141
q-big wrote: post #94

Earlier quoted context omitted.

> At least some of the control group will be in old age but still alive when we know it woris. The anti-aging serum could work (i.e. make you older), but have strong negative side effects.

So then it wouldn't "work", where "work" as used here is defined, "to function or operate according to plan or design". And no, it's not reasonable to assume I meant "work" as in, "have an anti-aging serum that has strong negative side effects."

An anti-aging serum works if it reduces or removed specific (or even most/all?) effects that are related to aging.

Re: Effect size is significantly more important than statistical significance

#146

Post by bluGill »

q-big wrote: post #145
TameAntelope wrote: post #141

Earlier quoted context omitted.

So then it wouldn't "work", where "work" as used here is defined, "to function or operate according to plan or design". And no, it's not reasonable to assume I meant "work" as in, "have an anti-aging serum that has strong negative side effects."

An anti-aging serum works if it reduces or removed specific (or even most/all?) effects that are related to aging.

Relative to other side effects. Remember we are in a post serum existing scenario so we would have the knowledge of those as part of the discussion on if it works.

Re: Effect size is significantly more important than statistical significance

#147

Post by bluGill »

EvgeniyZh wrote: post #132
jerf wrote: post #13

Earlier quoted context omitted.

I don't think we have found most of them. I think we make it look like we've found most of them because we keep throwing money at these crap studies. Bear in mind that my criteria are two-dimensional, and I'll accept either. By all means, go back and establish your 3% effect to a p-value of 0.0001. Or 0.000000001. That makes that 3% much more interesting and useful. It'll especially be interesting and valuable when y…

So 3% is not interesting but the difference between 10^-7 and 10^-8 probability that there is no effect is interesting somehow?

Meta analysis after enough small studies show the effect exists.

Re: Effect size is significantly more important than statistical significance

#148

Post by raxxorrax »

jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

The current science economy around publishing is partially responsible, although it should also be said that finding no correlation is still a gain of knowledge that is valuable to build upon for people in the same field, even if it might not generate the most exciting read for others.

Re: Effect size is significantly more important than statistical significance

#149

Post by pas »

neumann wrote: post #101
exporectomy wrote: post #9

I wonder if we should separate the roles of scientist and researcher. Universities would have generalist "scientists" who's job would be to consult for domain-specialized researchers to ensure they're doing the science and statistics correctly. That way, we don't need every researcher in every field to have a deep understanding of statistics, which they often don't. Either that or stop rewarding such bad behavior. Sc…

Every medical researcher I've worked with had a biostatistician on hand to handle the stats. As a aerospace engineer, I always had interesting discussions with them on the meaningfulness of a clinical study with 15 people, but have come to appreciate the massive difficulty in progressing medical research if everybody were to wait for a clinical trial with a 1000 patients.

There's no problem with a n=15 study, the problem is that there isn't a proper effective process that aggregates these small studies and the designs and conducts n=1000 ones. What we have instead is academic peacocking. (Grant applications judged by other scientists who are also at the same time in the grant game.)

Of course this is somewhat a necessary consequence of having academic freedom.

Re: Effect size is significantly more important than statistical significance

#150

Post by naasking »

kuhewa wrote: post #127
naasking wrote: post #120

Earlier quoted context omitted.

Such as? It would also be interesting to explore how many dead ends we wouldn't have wasted time on, and so what other things might have been discovered sooner.

Scientists aren't stupid. No one saw a paper where a predictor explained 1% of the variance in an outcome and based solely on a significant p value decided that was a great road to base an entire career on. The problem, as described by the parent comment, doesn't really exist in funding structures and the scientific literature. It does occur to some degree in media coverage of science. One could make the case that in…

> Scientists aren't stupid. No one saw a paper where a predictor explained 1% of the variance in an outcome and based solely on a significant p value decided that was a great road to base an entire career on. The problem, as described by the parent comment, doesn't really exist in funding structures and the scientific literature.

Of course they're stupid. Everyone is stupid. That's why we have a "scientific method" and a formal discipline of logic to overcome fallacious reasoning and cognitive biases. If people weren't stupid we wouldn't need any of these disciplines to check our mistakes.

And yes, what you describe does happen all of the time. We literally just had a thread on HN about the failure of the amyloid hypothesis in Alzheimer's and the decades of work put wasted on it. Many researchers are still trying to push it as a legitimate therapeutic target despite every clinical trial to date failing spectacularly. As Planck said, science advances on funeral at a time.

Which isn't to say that small effect sizes aren't legitimate research targets either, but if you're after a a small effect size, the rigour should be scaled proportionally.

Post reply on HN