Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

161–168 of 168 posts

Re: Effect size is significantly more important than statistical significance

#161

Post by 8note »

BoiledCabbage wrote: post #153
MaulingMonkey wrote: post #51

Earlier quoted context omitted.

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Mi…

That's a nice post and all, but none of that had anything to do with reliability. In all of those cases, those components worked exactly as designed when operating within their specification ranges (ie inserters insert when they have power). The point is, it would be significantly more complex if things frequently failed even when "operating properly". And this happened at all levels of abstraction in a factory.

It's still reliability, just who the whole system rather than the individual parts. The aliens breaking stuff is part of the whole system "operating properly"

I don't think it would be particularly bad for inserters inserting at slightly different speeds from each other, or occasionally destroying the item it was supposed to insert. Same with components occasionally breaking on their own.

Re: Effect size is significantly more important than statistical significance

#162

Post by kuhewa »

naasking wrote: post #150
kuhewa wrote: post #127

Earlier quoted context omitted.

Scientists aren't stupid. No one saw a paper where a predictor explained 1% of the variance in an outcome and based solely on a significant p value decided that was a great road to base an entire career on. The problem, as described by the parent comment, doesn't really exist in funding structures and the scientific literature. It does occur to some degree in media coverage of science. One could make the case that in…

> Scientists aren't stupid. No one saw a paper where a predictor explained 1% of the variance in an outcome and based solely on a significant p value decided that was a great road to base an entire career on. The problem, as described by the parent comment, doesn't really exist in funding structures and the scientific literature. Of course they're stupid. Everyone is stupid . That's why we have a "scientific method"…

So your example of decades being wasted chasing an initial tiny effect size, all the time, was... An example of a failed mechanistic hypothesis that wasn't based on a tiny effect size.

Re: Effect size is significantly more important than statistical significance

#163

Post by MaulingMonkey »

BoiledCabbage wrote: post #153
MaulingMonkey wrote: post #51

Earlier quoted context omitted.

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Mi…

That's a nice post and all, but none of that had anything to do with reliability. In all of those cases, those components worked exactly as designed when operating within their specification ranges (ie inserters insert when they have power). The point is, it would be significantly more complex if things frequently failed even when "operating properly". And this happened at all levels of abstraction in a factory.

You're drawing what appear to be arbitrary distinctions between failure modes without making a good argument as to why one is a reliability issue and another is not.

My printer might jam if I feed paper crooked or poorly. My assemblers might jam if I feed incorrect components through misclicks, misplaced miners, or filled outputs.

My printer might fail from the entropy of wear and tear. My assemblers might fail from the entropy of biters attracted by generated pollution.

My printer might stall from running out of paper or a filled output tray. My assemblers might stall from running out of inputs or a filled output belt or chest.

Why is the printer arguably unreliable, but the assembler "100% reliable"?

Failures of my printer are not caused by magic faries sprinkling dice rolling pixie dust on my toner cartrige. Failures have physical causes. That factorio's assembler failures have modeled causes as well, instead of an arbitrary and magic dice roll, does not detract from those failure modes being reliability issues.

That my printer fails far less frequently than my Factorio assemblers points to my printer being more reliable than my Factorio assemblers. Your point that reliability could be even worse misses my point, which is merely that not only does Factorio already avoid the fiction of "100%" or "perfect reliability" - but that perhaps Factorio already models reliability worse than "real-life" in some aspects already.

Re: Effect size is significantly more important than statistical significance

#164

Post by PeterisP »

hprotagonist wrote: post #157
harry8 wrote: post #134

Earlier quoted context omitted.

we tried using 0.10 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.13 mL, it didn't work we tried using 0.15 mL, it didn't work we tried using 0.17 mL, it didn't work we tried using 0.16 mL, it didn't work we tried using 0.18 mL, it didn't work we tried using 0.20 mL, it didn't work we tried using 0.14 mL, it didn't work we tried using 0.12 mL, it worked so we published Do you want to know…

i don’t want to know about each test that didn’t work as a separate publication, that’s for sure!

You would read a meta-study that summarizes those tests - especially because they might potentially made by different labs, and the fact that one of them worked might be actual a real effect caused by some other difference in the experiment.

Re: Effect size is significantly more important than statistical significance

#165

Post by naasking »

kuhewa wrote: post #162
naasking wrote: post #150

Earlier quoted context omitted.

> Scientists aren't stupid. No one saw a paper where a predictor explained 1% of the variance in an outcome and based solely on a significant p value decided that was a great road to base an entire career on. The problem, as described by the parent comment, doesn't really exist in funding structures and the scientific literature. Of course they're stupid. Everyone is stupid . That's why we have a "scientific method"…

So your example of decades being wasted chasing an initial tiny effect size, all the time, was... An example of a failed mechanistic hypothesis that wasn't based on a tiny effect size.

I wasn't trying to post about the effect size specifically, but about general incentives and dead ends, but if you want a specific example look no further than aspirin for myocardial infarction:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/

Quote:

> A commonly cited example of this problem is the Physicians Health Study of aspirin to prevent myocardial infarction (MI).4 In more than 22 000 subjects over an average of 5 years, aspirin was associated with a reduction in MI (although not in overall cardiovascular mortality) that was highly statistically significant: P Long-term aspirin use has its own risks, like GI bleeds, and the MI benefits are clearly not warranted given those risks.

Re: Effect size is significantly more important than statistical significance

#166

Post by harry8 »

harry8 wrote: post #134
hprotagonist wrote: post #108

Earlier quoted context omitted.

we tried using 0.1 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.12 mL, it didn't work we tried using 0.13 mL, it didn't work

we tried using 0.10 mL, it didn't work we tried using 0.11 mL, it didn't work we tried using 0.13 mL, it didn't work we tried using 0.15 mL, it didn't work we tried using 0.17 mL, it didn't work we tried using 0.16 mL, it didn't work we tried using 0.18 mL, it didn't work we tried using 0.20 mL, it didn't work we tried using 0.14 mL, it didn't work we tried using 0.12 mL, it worked so we published Do you want to know…

Especially for small effect size and suppressing what didn't work, this is one obvious way of many to perform p-hacking for publication acceptance.

https://en.wikipedia.org/wiki/Replication_crisis

Re: Effect size is significantly more important than statistical significance

#167

Post by robbedpeter »

exporectomy wrote: post #130
vkou wrote: post #68

Earlier quoted context omitted.

Nobody needs to caricature the insane beliefs surrounding COVID (or flat earth), people holding them are doing a good enough job of that themselves. I do have a few favorites. "COVID tests give you COVID, so I won't go get tested" is certainly up there. I can't say I give two figs about your opinion on the Earth's topology, but this one is a public health problem, that's crippling hospitals around the country.

So it didn't happen?

Exactly - like the "Oklahoma horse paste overdoses overwhelming emergency rooms so gunshot victims can't get tratment" the stories validate all the caricatured biases against middle America as a bunch of ignorant redneck yokels. None of the story was true, but people leapt on it because it resonates in that echo chamber.

Re: Effect size is significantly more important than statistical significance

#168

Post by kuhewa »

naasking wrote: post #165
kuhewa wrote: post #162

Earlier quoted context omitted.

So your example of decades being wasted chasing an initial tiny effect size, all the time, was... An example of a failed mechanistic hypothesis that wasn't based on a tiny effect size.

I wasn't trying to post about the effect size specifically, but about general incentives and dead ends, but if you want a specific example look no further than aspirin for myocardial infarction: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/ Quote: > A commonly cited example of this problem is the Physicians Health Study of aspirin to prevent myocardial infarction (MI).4 In more than 22 000 subjects over an av…

It's hard to parse that example, because the citation it contains is to a meta-analysis that provides and effect size of aspirin for MI in the PHS in the form of an odds ratio that is much greater magnitude. Digging a bit more, heres the actual result - the difference in relative risk was 44% not 0.77%. https://www.nejm.org/doi/full/10.1056/NEJM198907203210301

> There was a 44 percent reduction in the risk of myocardial infarction (relative risk, 0.56; 95 percent confidence interval, 0.45 to 0.70; PI agree if you said from the start you meant general incentives, especially in pharma development, but that is by and large a different conversation.

Post reply on HN