Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to avoid P hacking

nature.com

Post reply on HN
Print view View original

41–50 of 91 posts

Re: How to avoid P hacking

#41

Post by daveFNbuck »

pizlonator wrote: post #25
jean_lannes wrote: post #15

Earlier quoted context omitted.

These seem like two different things. Testing many different optimizations is not the same experiment; it's many different experiments. The SE equivalent of the practice being described would be repeatedly benchmarking code without making any changes and reporting results only from the favorable runs.

Doesn’t matter if it’s the same experiment or not. Say I’m after p<0.05. That means that if I try 40 different purported optimizations that are all actually neutral duds, one of them will seem like a speedup and one of them will seem like a slowdown, on average.

That's not p hacking. That's just the nature of p values. P hacking is when you do things to make a particular experiment more likely to show as a success.

Re: How to avoid P hacking

#42

Post by bbertelsen »

pizlonator wrote: post #11

The worst part about this: > Running experiments until you get a hit Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. Hence we are running experiments until we get a hit. The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn…

There's another cheeky example of this where you select a pseudo-random seed that makes your result significant. I have a personal seed, I use it in every piece of research that uses random number generation. It keeps me honest!

Re: How to avoid P hacking

#43

Post by LegionMammal978 »

starspangled wrote: post #31
pizlonator wrote: post #11

The worst part about this: > Running experiments until you get a hit Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. Hence we are running experiments until we get a hit. The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn…

> Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. I don't think that is what it is saying. It is saying you would write one particular optimization (your hypothesis), and then you would run the experiment (measuring speed-up) multiple times until you see a good number. It's fine to keep trying more optimiz…

"Multiple times" doesn't have to mean "no modifications". Suppose the software is currently on version A. You think that changing it to a version B might make it more performant, so you implement and profile it. You find no difference, so you figure that your B implementation isn't good enough, and write a slight variation B', perhaps moving around some loops or function calls. If that makes no difference, you keep writing variations B'', B''', B'''', etc., until one of them finally comes out faster than version A. You finally declare that version B (when properly implemented) is better than version A, when you've really just tried a lot more samples.

Re: How to avoid P hacking

#45

Post by nialse »

coolcase wrote: post #27
gwerbret wrote: post #23

> Stopping an experiment once you find a significant effect but before you reach your predetermined sample size is classic P hacking. Although much of the article is basic common sense, and although I'm not a statistician, I had to seriously question the author's understanding of statistics at this point. The predetermined sample size (statistical power) is usually based on an assumption made about the effect size; i…

Sounds like a variable cost experiment. Each observation cost x$. Like an A/B split on Google ads. Why keep paying for A when you know B is better already.

Small samples have more variability than large samples and thus more often show spurious large effects.

Re: How to avoid P hacking

#46

Post by remus »

spinf97 wrote: post #44

> Ending the experiment too early > Running experiments until you get a hit But if I'm running an experiment how do I know how many time to run it.

Before you start your experiment, you calculate how many samples you need based on the estimated effect size you're looking for and how small you want your confidence interval to be.

Small effect with high confidence => more samples

Big effect with low confidence=> less samples

Re: How to avoid P hacking

#47

Post by starspangled »

LegionMammal978 wrote: post #43
starspangled wrote: post #31

Earlier quoted context omitted.

> Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. I don't think that is what it is saying. It is saying you would write one particular optimization (your hypothesis), and then you would run the experiment (measuring speed-up) multiple times until you see a good number. It's fine to keep trying more optimiz…

"Multiple times" doesn't have to mean "no modifications". Suppose the software is currently on version A. You think that changing it to a version B might make it more performant, so you implement and profile it. You find no difference, so you figure that your B implementation isn't good enough, and write a slight variation B', perhaps moving around some loops or function calls. If that makes no difference, you keep w…

Well it does mean "no modifications" to the hypothesis, hypothesis being about performance of code A and B. Code B' would be a change.

It's just semantics, but the point is that the article wasn't saying the same thing OP was worried about. There's nothing wrong with testing B, B', B'', etc. until you find a significant performance improvement. You just wouldn't test B several times and take the last set of data when it looks good. Almost goes without saying really.

Re: How to avoid P hacking

#48

Post by neilv »

boulos wrote: post #19
neilv wrote: post #3

> As any gambler knows, if you roll the dice often enough, eventually you’ll get the result you want by chance alone You never count your results, when you're sitting at the lab bench, there will be time enough for counting, when the experiments are done.

Nicely done. Since many folks may not know the original song: https://en.m.wikipedia.org/wiki/The_Gambler_(song) (And TIL, this wasn't original to Kenny Rogers!)

I almost did this verbatim quote of the lyrics, which paralleled the article's sentence, and is relevant to P-hacking, but it's the wrong advice:

    Every gambler knows
    That the secret to survivin'
    Is knowin' what to throw away
    And knowin' what to keep

Re: How to avoid P hacking

#49

Post by coolcase »

nialse wrote: post #45
coolcase wrote: post #27

Earlier quoted context omitted.

Sounds like a variable cost experiment. Each observation cost x$. Like an A/B split on Google ads. Why keep paying for A when you know B is better already.

Small samples have more variability than large samples and thus more often show spurious large effects.

So you end up with a higher threshold for confidence at pToss a coin 10 times comes up heads 10 times. There is a 1 in 2^10 (approx 1000) that happens by chance for an unbiased coin.

I'm convinced it is biased.

20 times I am freaking convinced.

I don't need another 1000 tosses.

Re: How to avoid P hacking

#50

Post by AstralStorm »

parpfish wrote: post #30

I was heavily encouraged to do what would later be called “p-hacking”, but it looked different from what they describe here. This article describes p-hacks for people that aren’t into math/stats. I always ended up p hacking because I was into stats methods. Somebody would say “here’s an old dataset that didn’t work out, I bet you can use one of those new stats methods you’re always reading about to find a cool effect…

The practice you describe is called data dredging though. The thing about it is that you do not know enough experimental design details to make sure it was all on the up, especially worse the older the dataset gets.

Normally when doing that you need a multiple comparison corrections and conservative stats. That won't get you published though, or if you do get published you won't get noticed except by someone running a meta analysis. Perhaps not even then. Usually you end up with negative results from reanalysis, evidence of tampering or small effect sizes.

And this does not that reliably detect dataset manipulation, p hacking on the part of experimenters or accidental violations of the protocol, not even necessarily if the data collection included measures to prevent it.

In short: you cannot 100% trust any dataset you did not make. Not even as part of the team that makes it.

Post reply on HN