Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to avoid P hacking

nature.com

Post reply on HN
Print view View original

11–20 of 91 posts

Re: How to avoid P hacking

#11

Post by pizlonator »

The worst part about this:

> Running experiments until you get a hit

Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up.

Hence we are running experiments until we get a hit.

The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn't see the speed-up, then you just p-hacked yourself. But that's not even fool proof.

You just have to accept that statistics lie and that you will fool yourself. Prepare accordingly.

Re: How to avoid P hacking

#13

Post by smallmancontrov »

nmca wrote: post #9
smallmancontrov wrote: post #4

It might be below the fold, but it looks like they're missing the most important p-hacking strategy of all: the dogshit null hypothesis. It's very reliable and it's the most common type of p-hacking that I see. It's easy to create a dogshit null hypotheses by negligence or by "negligence" and it's easy to reject a dogshit null hypothesis by simply collecting enough data as it automatically crumbles on contact with th…

This would be much better with an example

"I ran a t-test on the untreated / treated samples and the difference is significant! The treatment worked!"

...but the data table shows a clear trend over time across both groups because the samples were being irradiated by intense sunlight from a nearby window. The model didn't account for this possibility, so it was rejected, just not because the treatment worked.

That's a relatively trivial example and you can already imagine ways in which it could have occurred innocently and not-so-innocently. Most of the time it isn't so straightforward. The #1 culprit I see is failure to account for some kind of obvious correlation, but the ways in which a null hypothesis can be dogshit are as numerous and subtle as the number of possible statistical modeling mistakes in the universe because they are the same thing.

Re: How to avoid P hacking

#14

Post by throwanem »

pizlonator wrote: post #11

The worst part about this: > Running experiments until you get a hit Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. Hence we are running experiments until we get a hit. The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn…

Why is this bad for you? You're optimizing software, not trying to describe reality. Monte Carlo and Drunkard's Walk are fine.

Re: How to avoid P hacking

#15

Post by jean_lannes »

pizlonator wrote: post #11

The worst part about this: > Running experiments until you get a hit Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. Hence we are running experiments until we get a hit. The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn…

These seem like two different things. Testing many different optimizations is not the same experiment; it's many different experiments. The SE equivalent of the practice being described would be repeatedly benchmarking code without making any changes and reporting results only from the favorable runs.

Re: How to avoid P hacking

#16

Post by analog31 »

throwanem wrote: post #14
pizlonator wrote: post #11

The worst part about this: > Running experiments until you get a hit Is that it's literally what us software optimization engineers do. We keep writing optimizations until we find one that is a statistically significant speed-up. Hence we are running experiments until we get a hit. The only defense I know against this is to have a good perf CI. If your patch seemed like a speed-up before committing, but perf CI doesn…

Why is this bad for you? You're optimizing software, not trying to describe reality. Monte Carlo and Drunkard's Walk are fine.

You're churning the user experience for no reason. Maybe constant optimization churn is one of the reasons why UIs are so bad.

Re: How to avoid P hacking

#17

Post by analog31 »

p4ul wrote: post #2

If the conclusion is "be transparent", I'm strongly supportive. And moreover, I would be even more supportive if we found a way to change the incentives for tenure and promotion such that reproducibility was an important factor in how we make decisions about grants, tenure, and promotion.

Just make it even more cutthroat than it already is. Replacing one hackable incentive system with another will just produce a new set of hacks.

Disclosure: I left academia before I had to worry about any of this.

Re: How to avoid P hacking

#18

Post by somenameforme »

smallmancontrov wrote: post #13
nmca wrote: post #9

Earlier quoted context omitted.

This would be much better with an example

"I ran a t-test on the untreated / treated samples and the difference is significant! The treatment worked!" ...but the data table shows a clear trend over time across both groups because the samples were being irradiated by intense sunlight from a nearby window. The model didn't account for this possibility, so it was rejected, just not because the treatment worked. That's a relatively trivial example and you can al…

I think you're more observing an issue with experimental models not challenging a null hypothesis, than with poor null hypotheses themselves. In other words, papers creating experiments that don't actually challenge the hypothesis. There was a major example of this with COVID. A typical way observational studies assessed the efficacy of the vaccines was by looking at outcomes between normalized samples of nonvaccinated and vaccinated individuals who came to the hospital and seeing their overall outcomes. Unvaccinated individuals generally had worse outcomes, so therefore the vaccines must be effective.

This logic was used repeatedly, but it fails to account for numerous obvious biases. For instance unvaccinated people are generally going to be less proactive in seeking medical treatment, and so the average severity of a case that causes them to go to the hospital is going to be substantially greater than for a vaccinated individual, with an expectation of correspondingly worse overall outcomes. It's not like this is some big secret - most papers mentioned this issue (among many others) in the discussion, but ultimately made no effort to control for it.

Re: How to avoid P hacking

#19

Post by boulos »

neilv wrote: post #3

> As any gambler knows, if you roll the dice often enough, eventually you’ll get the result you want by chance alone You never count your results, when you're sitting at the lab bench, there will be time enough for counting, when the experiments are done.

Nicely done. Since many folks may not know the original song: https://en.m.wikipedia.org/wiki/The_Gambler_(song)

(And TIL, this wasn't original to Kenny Rogers!)

Post reply on HN