Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

111–120 of 168 posts

Re: Effect size is significantly more important than statistical significance

#111

Post by captain_price7 »

candiodari wrote: post #82
DiabloD3 wrote: post #66

Earlier quoted context omitted.

Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.

You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.

Why would someone want to game a negative result? Nobody ever becomes famous for saying my approach doesn't work. (As long as science is open, to make sure there is actually good work done by researchers before reaching this neg result.)

Re: Effect size is significantly more important than statistical significance

#112

Post by jrk »

chuckcode wrote: post #98
jerf wrote: post #2

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to…

Don't get distracted by the click bait title. Effect size should be captured by statistical significance (larger effects are less likely to happen by chance). Author is really complaining that the original study didn't report enough data to check their analysis or do alternative analysis methods. Better title for article would be "Hard to peer review when you don't share the data"

Note the point in the essay that statistical significance is meaningless if the model does not correspond to reality — which, in this case as in many, they very much do not.

Re: Effect size is significantly more important than statistical significance

#113

Post by nabla9 »

jiggunjer wrote: post #109
nabla9 wrote: post #47

If you have one BALB/c lab mouse, you give it something, and it glows in the dark few months after, effect size alone makes it significant.

there is no effect size if you have no control group. so you'd need 2 mice at the least.

Only if you have not seen lab mouse before.

Re: Effect size is significantly more important than statistical significance

#114

Post by mbrodersen »

reilly3000 wrote: post #54
MaulingMonkey wrote: post #51

Earlier quoted context omitted.

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Mi…

I really considered getting into Factorio but your comment is exactly why I can’t touch it. I have certain demands upon my time that would inevitably go unmet as I fuss with factory.

Yes be warned. Factorio is the most addictive game I have ever played.

Re: Effect size is significantly more important than statistical significance

#115

Post by Ice_cream_suit »

The better medical journals do stress the hazard ration efficacy and the confidence interval.

See the extract below from the NEJM: Seasonal Malaria Vaccination with or without Seasonal Malaria Chemoprevention

"The hazard ratio for the protective efficacy of RTS,S/AS01E as compared with chemoprevention was 0.92 (95% confidence interval [CI], 0.84 to 1.01), which excluded the prespecified noninferiority margin of 1.20.

The protective efficacy of the combination as compared with chemoprevention alone was 62.8% (95% CI, 58.4 to 66.8) against clinical malaria, 70.5% (95% CI, 41.9 to 85.0) against hospital admission with severe malaria according to the World Health Organization definition, and 72.9% (95% CI, 2.9 to 92.4) against death from malaria.

The protective efficacy of the combination as compared with the vaccine alone against these outcomes was 59.6% (95% CI, 54.7 to 64.0), 70.6% (95% CI, 42.3 to 85.0), and 75.3% (95% CI, 12.5 to 93.0), respectively."

https://www.nejm.org/doi/full/10.1056/NEJMoa2026330?query=fe...

Re: Effect size is significantly more important than statistical significance

#117

Post by divbzero »

DiabloD3 wrote: post #66
naasking wrote: post #40

Earlier quoted context omitted.

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.

I like rules like these. One context where preregistration, null results, and full data are all required are clinical trials overseen by the FDA. It’s no surprise that those studies carry a lot of weight.

Re: Effect size is significantly more important than statistical significance

#118

Post by NavinF »

hprotagonist wrote: post #108
naasking wrote: post #92

Earlier quoted context omitted.

Yes, and knowing what's been tried and what has failed is important.

we tried using 0.1 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.12 mL, it didn't work we tried using 0.13 mL, it didn't work

If someone really tested those hypotheses, let them publish. I doubt they'll get funding so it'll be on their own dime. In practice people do run experiments like that, but they only publish the 1/4 trials that is successful.

Re: Effect size is significantly more important than statistical significance

#119

Post by naasking »

hprotagonist wrote: post #108
naasking wrote: post #92

Earlier quoted context omitted.

Yes, and knowing what's been tried and what has failed is important.

we tried using 0.1 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.12 mL, it didn't work we tried using 0.13 mL, it didn't work

Look to physics for how negative results should be published. There typically has to be reason to suspect some dosage range should work, in which case that sequence of studies you describe would be perfectly valid if it's within that range.

Re: Effect size is significantly more important than statistical significance

#120

Post by naasking »

Angostura wrote: post #96
naasking wrote: post #40

Earlier quoted context omitted.

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

It would be interesting to consider how much knowledge would never have been uncovered if you were King of Science. All those subtle, barely seen interactions in nature that on further investigation turned out to be something rather special.

Such as? It would also be interesting to explore how many dead ends we wouldn't have wasted time on, and so what other things might have been discovered sooner.
Post reply on HN