Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

p-Hacking and False Discovery in A/B Testing

papers.ssrn.com

Post reply on HN
Print view View original

21–30 of 106 posts

Re: p-Hacking and False Discovery in A/B Testing

#21

Post by boron1006 »

aisscott wrote: post #15

Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper. If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://w…

I'm curious about the wording "effects are truly null". I was always under the impression that you can never really "accept the null", but rather "fail to reject the alternative".

Re: p-Hacking and False Discovery in A/B Testing

#22

Post by cepth »

Fomite wrote: post #18
hammock wrote: post #16

Earlier quoted context omitted.

This is wrong. P hacking has nothing to do with the p being too lenient.

This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.

> This is wrong. P hacking has nothing to do with the p being too lenient.

> This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.

I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic.

P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to address the lack of reproducibility is not something that I made up. Yes, lowering the p-value threshold does not eliminate the motivations/techniques that are necessary for p-hacking, but it can make it a lot harder, and a lot less worthwhile. If you work in academia, and it takes you much longer to now cherrypick a sample to meet a much lower p-cutoff, it seems to follow that we would see less of it.

This is an excerpt from a paper soon to be published in Nature: https://imai.princeton.edu/research/files/significance.pdf. The key quote: 'We have diverse views about how best to improve reproducibility,and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.'

With regards to the comment that you could p-hack up to any strictness, I'm not sure this is correct. If you accept the proposal laid out in that Nature paper, to lower the threshold to P<0.005, or if we go even lower to P<0.001 I don't believe that you'd be able to p-hack in any practical way. Yes, you could cherry pick a tiny sample, but any peer reviewer or colleague of yours is going to ask questions about the sample.

Re: p-Hacking and False Discovery in A/B Testing

#23

Post by nanis »

Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts.

I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result."

I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a thing.

Re: p-Hacking and False Discovery in A/B Testing

#24

Post by dahdum »

After listening to Optimizely reps give a talk about their success with a client (who was present) I suspect the support reps encourage these false positives. They presented a few tests as fantastic wins, when they all had basic flaws (like cold audience vs self selected for interest). Maybe that was just one bad apple (doubtful)...but it was a large client and someone they felt should represent the company as a speaker.

Concerns from the audience were dismissed and referred to follow up after the talk. Never thought the same of Optimizely after that.

Re: p-Hacking and False Discovery in A/B Testing

#25

Post by Fomite »

cepth wrote: post #22
Fomite wrote: post #18

Earlier quoted context omitted.

This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.

> This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to addre…

I'm not being nitpicky - they are components of a problem of reproducibility, but orthogonal to each other. Bad UI design and a poor backend are both reasons "X website sucks!" but that doesn't mean they're the same.

A perfectly designed, un-p-hacked study should still perhaps be held to a stricter p-value criteria than 0.05.

And I am correct - because I've done it. Presently working on a paper where, because I primarily work with simulations, I can translate minute and meaningless difference into arbitrarily small p-values. And I used arbitrary for a good reason - my personal record is the smallest value R can express.

Ironically, this isn't because I have a tiny sample, but because I can make tremendously large ones. All of this is because no where in the calculation of a p-value is the question "Does this different matter?"

Re: p-Hacking and False Discovery in A/B Testing

#26

Post by windows_tips »

boron1006 wrote: post #21
aisscott wrote: post #15

Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper. If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://w…

I'm curious about the wording "effects are truly null". I was always under the impression that you can never really "accept the null", but rather "fail to reject the alternative".

The "null" hypothesis would typically be that "there is no effect".

All that you can really do is prove it wrong, by measuring an effect when there "should", by the hypothesis, be none.

Due to what is known as the "problem of induction", it's not sufficient to accept a hypothesis because you appeared to not measure an effect in the past, as that says nothing about whether an effect will occur the next time a measurement is made.

p-value is the "chance" of measuring an effect, given that no effect actually occurred.

Re: p-Hacking and False Discovery in A/B Testing

#28

Post by hammock »

cepth wrote: post #22
Fomite wrote: post #18

Earlier quoted context omitted.

This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.

> This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to addre…

The parent asked very straightforwardly what is p hacking and you replied with a red herring. If I'm being pedantic, you're being unhelpful.

Re: p-Hacking and False Discovery in A/B Testing

#29

Post by cepth »

Fomite wrote: post #25
cepth wrote: post #22

Earlier quoted context omitted.

> This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to addre…

I'm not being nitpicky - they are components of a problem of reproducibility, but orthogonal to each other. Bad UI design and a poor backend are both reasons "X website sucks!" but that doesn't mean they're the same. A perfectly designed, un-p-hacked study should still perhaps be held to a stricter p-value criteria than 0.05. And I am correct - because I've done it. Presently working on a paper where, because I prima…

First off, I don’t have any experience with publishing based off the results of simulations. My (short) time in writing papers centered around economics research with observational datasets.

I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way. My colleagues would’ve asked why the gigantic dataset we purchased had 1/4 of its observations thrown out etc.

Your claim that the threshold and the practice of p-hacking are orthogonal (independent?) is still puzzling to me. I think a better analogy would be trying to game something like your Pagespeed score. In order to get a higher score, you skimp on UX so the page loads faster, and cut out backend functionality because you want fewer HTTP requests. Making it harder to achieve a Pagespeed score forces you at some point to evaluate the tradeoffs of chasing that score.

I have two questions for you:

1) Would it take you more time to p-hack a lower threshold, or do all your results yield you a ~0.0000 p-value?

2) In simulation based research like yours, it seems to me that even other p-hacking “fixes” like forcing there to be a pre-announcement or sample size, sample structure, etc. wouldn’t address what you say you’re able to do. What can be done to fix it?

Re: p-Hacking and False Discovery in A/B Testing

#30

Post by cepth »

hammock wrote: post #28
cepth wrote: post #22

Earlier quoted context omitted.

> This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to addre…

The parent asked very straightforwardly what is p hacking and you replied with a red herring. If I'm being pedantic, you're being unhelpful.

> clever experimental design and cherry picking can generate many results that are statistically significant at that level.

I’m not sure what’s incorrect about this statement? If you disagree with the “fix” to the problem that is most familiar to me, that’s fine. It’s one of many approaches.

But, at what point did I mislead the parent as to what p-hacking is? What’s your definition?

Post reply on HN