Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

91–100 of 168 posts

Re: Effect size is significantly more important than statistical significance

#91

Post by jltsiren »

exporectomy wrote: post #9

I wonder if we should separate the roles of scientist and researcher. Universities would have generalist "scientists" who's job would be to consult for domain-specialized researchers to ensure they're doing the science and statistics correctly. That way, we don't need every researcher in every field to have a deep understanding of statistics, which they often don't. Either that or stop rewarding such bad behavior. Sc…

Such staff scientist roles for people with particular methodological skills do exist. They are not particularly common, because there are a few issues:

1. Who will pay for them?

2. How do we make staff scientist roles attractive to people who could also get tenure-track faculty positions or do ML/data science in the industry?

3. How do we ensure that a staff scientist position is not a career dead end if the funding dries up after a decade or two?

The standard academic incentives (long-term stability provided by tenure, freedom to work on whatever you find interesting, recognition among other experts in the field) don't really apply to support roles.

Re: Effect size is significantly more important than statistical significance

#92

Post by naasking »

candiodari wrote: post #82
DiabloD3 wrote: post #66

Earlier quoted context omitted.

Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.

You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.

Yes, and knowing what's been tried and what has failed is important.

Re: Effect size is significantly more important than statistical significance

#93

Post by Karrot_Kream »

sdenton4 wrote: post #84
Karrot_Kream wrote: post #73

Earlier quoted context omitted.

> Which rate? The rate you failed to mix the balls? The rate you failed to count a ball? The rate you misclassified the ball? The rate you repeatedly counted the same ball? The rate you started with an incorrect count? The rate you did the math wrong? etc This is called modelling error. Both Bayesian and frequentist approaches suffer from modelling error. That's what TFA talks about when mentioning the normality assu…

I wouldn't think of Black Swan events as tail events, so much as model failures or regime-changes. As in, 'we modeled this as a time-invariant gaussian distribution, but it's actually a mixture model where the second hidden mode was triggered in the aftermath of an asteroid strike that we didn't model for, because of course we didn't.' In re, the arguey-person you were responding to, frequentist modeling is just as b…

Ah yeah fair enough, I see what you mean. This is a general problem with all models though. Fundamental modeling issues will tank your conclusion.

Re: Effect size is significantly more important than statistical significance

#94

Post by q-big »

bluGill wrote: post #78
q-big wrote: post #71

Earlier quoted context omitted.

> Nobody likes being in the control group of the first working anti-aging serum... You only know whether it works when the study has been completed. You also only know whether the drug has (potentially) disastrous consequences when the study has been completed. Thus, I am not completely sure whether your claim holds.

You missed the working part. Success was a prerequisite to their after the fact feelings. At least some of the control group will be in old age but still alive when we know it woris. They might not know if it is infinite life (and side effects may turn it into die at 85, so some control may outlive the intervention group after the study ), but they will know on average they did worse

> At least some of the control group will be in old age but still alive when we know it woris.

The anti-aging serum could work (i.e. make you older), but have strong negative side effects.

Re: Effect size is significantly more important than statistical significance

#95

Post by [deleted] »

ummonk wrote: post #77

Agree with the title, but not the contents. The study in question is actually an example of a huge effect size (10% reduction in cases just from instructing villages they should wear masks is amazing) possibly hampered by poor statistical significance (as the blog post outlines).

[deleted]

Re: Effect size is significantly more important than statistical significance

#96

Post by Angostura »

naasking wrote: post #40
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

It would be interesting to consider how much knowledge would never have been uncovered if you were King of Science. All those subtle, barely seen interactions in nature that on further investigation turned out to be something rather special.

Re: Effect size is significantly more important than statistical significance

#97

Post by HWR_14 »

q-big wrote: post #71
TameAntelope wrote: post #37

Earlier quoted context omitted.

Nobody likes being in the control group of the first working anti-aging serum...

> Nobody likes being in the control group of the first working anti-aging serum... You only know whether it works when the study has been completed. You also only know whether the drug has (potentially) disastrous consequences when the study has been completed. Thus, I am not completely sure whether your claim holds.

People opt into the study in the first place. I'm willing to bet that no one opts into the study hoping to be in the control group.

Re: Effect size is significantly more important than statistical significance

#98

Post by chuckcode »

jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

Don't get distracted by the click bait title. Effect size should be captured by statistical significance (larger effects are less likely to happen by chance). Author is really complaining that the original study didn't report enough data to check their analysis or do alternative analysis methods. Better title for article would be "Hard to peer review when you don't share the data"

Re: Effect size is significantly more important than statistical significance

#99

Post by [deleted] »

spywaregorilla wrote: post #74
_Nat_ wrote: post #56

The title's misinformation: effect-size ISN'T more important than statistical significance. The article itself makes some better points, e.g. > I worry that because of statistical ambiguity, there’s not much that can be deduced at all. , which would seem like a reasonable interpretation of the study that the article discusses. However, the title alone seems to assert a general claim about statistical interpretation t…

Not so fast. If you win your first jackpot on the first ticket. You'll require 500,000 failures (at $1 per ticket) in order to fail to reject the null hypothesis at p If you bought just ten tickets you would have a p value below 0.0000001 And that makes sense, because a p value of 0.01 says the probability of getting a sample this far from the null hypothesis is less than 1 in a million by random chance... which is w…

[deleted]

Re: Effect size is significantly more important than statistical significance

#100

Post by reificator »

mercurywells wrote: post #46
Robotbeat wrote: post #7

Earlier quoted context omitted.

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it…

> I’ve been thinking about this while playing Factorio: so much of our discussion and mental modeling of automation works under the assumption of perfect reliability. If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. So I'm making a guess here that you play with few monsters or non-aggressive monsters?

> So I'm making a guess here that you play with few monsters or non-aggressive monsters?

Aggressively building turret walls, defensive train lines, and so on very quickly pays dividends here. Particularly if you claim as much territory as you can each time you expand instead of simply defending what you've built out.

If done this way building/improving defenses and managing enemies becomes a task you maintain every so often and doesn't spill over into the reliability of your base.

Post reply on HN