Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effect size is significantly more important than statistical significance

argmin.net

Post reply on HN
Print view View original

101–110 of 168 posts

Re: Effect size is significantly more important than statistical significance

#101

Post by neumann »

exporectomy wrote: post #9

I wonder if we should separate the roles of scientist and researcher. Universities would have generalist "scientists" who's job would be to consult for domain-specialized researchers to ensure they're doing the science and statistics correctly. That way, we don't need every researcher in every field to have a deep understanding of statistics, which they often don't. Either that or stop rewarding such bad behavior. Sc…

Every medical researcher I've worked with had a biostatistician on hand to handle the stats. As a aerospace engineer, I always had interesting discussions with them on the meaningfulness of a clinical study with 15 people, but have come to appreciate the massive difficulty in progressing medical research if everybody were to wait for a clinical trial with a 1000 patients.

Re: Effect size is significantly more important than statistical significance

#102

Post by tomcam »

reilly3000 wrote: post #54
MaulingMonkey wrote: post #51

Earlier quoted context omitted.

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Mi…

I really considered getting into Factorio but your comment is exactly why I can’t touch it. I have certain demands upon my time that would inevitably go unmet as I fuss with factory.

Holy shit I was about to compose exactly this answer! Parent was marketing straight to my lizard brain

Re: Effect size is significantly more important than statistical significance

#103

Post by ghodith »

MaulingMonkey wrote: post #51
Robotbeat wrote: post #7

Earlier quoted context omitted.

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it…

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Mi…

Now imagine if machines got clogged 1% of the time and you had to fix them, or if items occasionally fell off conveyer belts onto other conveyor belts. The amount of redundancy and work that would create would be paralyzing, but that’s the bare minimum of recreating what goes wrong in the real world. I love factorio, but what always strikes me as most interesting is thinking about what it is you get to take for granted in one of the most complex games around.

Re: Effect size is significantly more important than statistical significance

#104

Post by falcor84 »

HWR_14 wrote: post #97
q-big wrote: post #71

Earlier quoted context omitted.

> Nobody likes being in the control group of the first working anti-aging serum... You only know whether it works when the study has been completed. You also only know whether the drug has (potentially) disastrous consequences when the study has been completed. Thus, I am not completely sure whether your claim holds.

People opt into the study in the first place. I'm willing to bet that no one opts into the study hoping to be in the control group.

Well, some medical studies pay good money, so it's perfectly rational to sign up for these and hope to get the placebo.

Re: Effect size is significantly more important than statistical significance

#105

Post by yesOfCourse9 »

Shock; understanding isn’t reached by merely mapping the particles; it’s particles and waves.

I feel like there is a fundamental philosophy to science we decided did not matter when we switched to “just compute it” effort.

Yin and Yang, particle/wave duality, matter and fundamental forces; weaved into human culture going back forever there is a commonly occurring meme one might distill as “nothing exists without the right balance”.

Re: Effect size is significantly more important than statistical significance

#106

Post by HWR_14 »

falcor84 wrote: post #104
HWR_14 wrote: post #97

Earlier quoted context omitted.

People opt into the study in the first place. I'm willing to bet that no one opts into the study hoping to be in the control group.

Well, some medical studies pay good money, so it's perfectly rational to sign up for these and hope to get the placebo.

Fair enough. I tend to think of most of them as paying just a small amount where the main goal is to get treatment, but I'm sure some pay better than others.

Re: Effect size is significantly more important than statistical significance

#107

Post by TOMDM »

naasking wrote: post #92
candiodari wrote: post #82

Earlier quoted context omitted.

You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.

Yes, and knowing what's been tried and what has failed is important.

I think what's being pointed out is that "researchers" could pump out hundreds of easy to test negatives every day if a negative result was just as incentivised.

I do agree though, negatives are just as important when the intent is to prove/disprove a meaningful hypothosis.

Re: Effect size is significantly more important than statistical significance

#108

Post by hprotagonist »

naasking wrote: post #92
candiodari wrote: post #82

Earlier quoted context omitted.

You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.

Yes, and knowing what's been tried and what has failed is important.

we tried using 0.1 mL, it didn't work

we tried using 0.11 mL, it didn't work

we tried using 0.12 mL, it didn't work

we tried using 0.13 mL, it didn't work

Re: Effect size is significantly more important than statistical significance

#109

Post by jiggunjer »

nabla9 wrote: post #47

If you have one BALB/c lab mouse, you give it something, and it glows in the dark few months after, effect size alone makes it significant.

there is no effect size if you have no control group. so you'd need 2 mice at the least.

Re: Effect size is significantly more important than statistical significance

#110

Post by mr_toad »

robocat wrote: post #30

From the article: Ernest Rutherford is famously quoted proclaiming “If your experiment needs statistics, you ought to have done a better experiment.” “Of course, there is an existential problem arguing for large effect sizes. If most effect sizes are small or zero, then most interventions are useless. And this forces us scientists to confront our cosmic impotence, which remains a humbling and frustrating experience.”

Must be nice. Not everyone has the luxury of being able to carry out whatever experimentation they feel like. Sometimes we’re limited by what is affordable, practical, or ethical.
Post reply on HN