Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to avoid P hacking

nature.com

Post reply on HN
Print view View original

51–60 of 91 posts

Re: How to avoid P hacking

#51

Post by AstralStorm »

bjornsing wrote: post #34
gwerbret wrote: post #23

> Stopping an experiment once you find a significant effect but before you reach your predetermined sample size is classic P hacking. Although much of the article is basic common sense, and although I'm not a statistician, I had to seriously question the author's understanding of statistics at this point. The predetermined sample size (statistical power) is usually based on an assumption made about the effect size; i…

There are of course statistical methods designed to support early stopping. But I don’t think you can use a regular p-test every day and decide to stop if p < 0.05. That’s something else.

You use full both sided ANOVA F test with multiple comparison correction for that. Even these tests are sometimes not conservative enough, because the correction is a bit of a guess.

You will end up with much higher number of trials required to hit the P value than the version with predetermined number of trials and no stopping point by P.

Say, in a single variable single run ABX test, 8 is the usual number needed according to Fischer frequentist approach. If you do multiple comparison to hit 0.05 you need I believe 21 trials instead. (Don't quote me on that, compute your own Bayesian beta prior probability.)

The number of trials to differentiate from a fair coin is the typical comparison prior, giving a beta distribution. You're trying to set up a ratio between the two of them, one fitted to your data, the other null.

Re: How to avoid P hacking

#52

Post by mjburgess »

hiddencost wrote: post #33
gwerbret wrote: post #23

> Stopping an experiment once you find a significant effect but before you reach your predetermined sample size is classic P hacking. Although much of the article is basic common sense, and although I'm not a statistician, I had to seriously question the author's understanding of statistics at this point. The predetermined sample size (statistical power) is usually based on an assumption made about the effect size; i…

https://commons.m.wikimedia.org/wiki/File:P-hacking_by_early... The author is absolutely correct. Early stopping is a classic form of p hacking. See attached image for an illustration. If you want to be rigorous, you can define criterion for early stopping such that it's not, but you require relatively stronger evidence. Clinical trials that stop early do so typically at predefined times with higher significance thre…

The region where `p` hits the red line should be called "publish or perish".

Re: How to avoid P hacking

#54

Post by [deleted] »

pizlonator wrote: post #11

The worst part about this: > Running experiments until you get a hit Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. Hence we are running experiments until we get a hit. The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn…

[deleted]

Re: How to avoid P hacking

#55

Post by dr_dshiv »

rrr_oh_man wrote: post #40
coolcase wrote: post #27

Earlier quoted context omitted.

Sounds like a variable cost experiment. Each observation cost x$. Like an A/B split on Google ads. Why keep paying for A when you know B is better already.

Google Optimize used to tell you to let an experiment run for one-two weeks (?), exactly because early strong results tend to not don't hold up in the long run. -> https://en.wikipedia.org/wiki/Regression_toward_the_mean

Seasonality effects, too

Re: How to avoid P hacking

#56

Post by thelamest »

AstralStorm wrote: post #51
bjornsing wrote: post #34

Earlier quoted context omitted.

There are of course statistical methods designed to support early stopping. But I don’t think you can use a regular p-test every day and decide to stop if p < 0.05. That’s something else.

You use full both sided ANOVA F test with multiple comparison correction for that. Even these tests are sometimes not conservative enough, because the correction is a bit of a guess. You will end up with much higher number of trials required to hit the P value than the version with predetermined number of trials and no stopping point by P. Say, in a single variable single run ABX test, 8 is the usual number needed ac…

The general topic and some specific ways to estimate a correction are described under this term: https://en.wikipedia.org/wiki/Sequential_analysis

Re: How to avoid P hacking

#57

Post by saghm »

neilv wrote: post #48
boulos wrote: post #19

Earlier quoted context omitted.

Nicely done. Since many folks may not know the original song: https://en.m.wikipedia.org/wiki/The_Gambler_(song) (And TIL, this wasn't original to Kenny Rogers!)

I almost did this verbatim quote of the lyrics, which paralleled the article's sentence, and is relevant to P-hacking, but it's the wrong advice: Every gambler knows That the secret to survivin' Is knowin' what to throw away And knowin' what to keep

I don't know, maybe knowing when to "hold them" versus "fold them" and "walk away" would be a valuable skill here. The phrasing sounds off in the part you quite because in poker you only can play a given hand once, and after you've lost, you need to draw an entirely new dataset and start fresh.

Re: How to avoid P hacking

#58

Post by zipy124 »

The Bonferroni correction part of this article is the most important. The amount of papers that don't account for this is shocking, comparing 20 variables with a 0.05 confidence interval is extremely annoying, as you end up having to do analysis on all papers data yourself to correct for it to see if it is still significant or not.

Re: How to avoid P hacking

#59

Post by azan_ »

coolcase wrote: post #49
nialse wrote: post #45

Earlier quoted context omitted.

Small samples have more variability than large samples and thus more often show spurious large effects.

So you end up with a higher threshold for confidence at p Toss a coin 10 times comes up heads 10 times. There is a 1 in 2^10 (approx 1000) that happens by chance for an unbiased coin. I'm convinced it is biased. 20 times I am freaking convinced. I don't need another 1000 tosses.

It’s more like you are supposed to toss 1000 times and after 500 tosses you get a lucky streak of 5 heads in a row and then decide to end experiment and conclude that coin is biased.

Re: How to avoid P hacking

#60

Post by pcrh »

gwerbret wrote: post #23

> Stopping an experiment once you find a significant effect but before you reach your predetermined sample size is classic P hacking. Although much of the article is basic common sense, and although I'm not a statistician, I had to seriously question the author's understanding of statistics at this point. The predetermined sample size (statistical power) is usually based on an assumption made about the effect size; i…

The distinction is between ‘data peeking’, i.e. repeatedly checking the p-value you've obtained and stopping if it falls below 0.05, and repeating assays in the light of new information. Such new information can relate to the distribution of the values, the expected effect size, or any other parameter that you did not know at the outset of the study.

In ‘data peeking’, the flaw is that if an assay is repeated often enough, one will eventually get a result that deviates far from the mean result. This is a natural consequence of the data having a normal distribution, i.e. not all results will be identical. It's the equivalent of getting six heads or tails in a row (which should happen at least once if you flip a coin 200 times), and then reporting your coin as biased.

Repeating an assay because the distribution of the data is not what you thought, or because the likely difference between means is smaller than you thought is a valid approach.

Source: Big little lies: a compendium and simulation of p-hacking strategies Angelika M. Stefan and Felix D. Schönbrodt

https://royalsocietypublishing.org/doi/10.1098/rsos.220346

Post reply on HN