Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

p-Hacking and False Discovery in A/B Testing

papers.ssrn.com

Post reply on HN
Print view View original

91–100 of 106 posts

Re: p-Hacking and False Discovery in A/B Testing

#91

Post by alexgmcm »

smallnamespace wrote: post #82
yichijin wrote: post #78

Earlier quoted context omitted.

Hey all, statistician from Optimizely chiming in here. Just wanted to point out that this is exactly the right point. I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple look…

What's the tradeoff vs. just taking a direct Bayesian approach? In fact, why use an inferential framework at all (estimating some sort of probability and using it to guide action), rather than directly using a policy learning framework, e.g. modeling this as Q-learning or multi-armed bandit problem? If at the end of the day you have some objective function (e.g. 'making money'), some known space of actions (e.g. move…

A policy learning approach is better imho - but getting people to switch to using a multi-armed bandit when they are used to AB testing can be difficult.

People don't seem to trust the system to make the right decisions even though you can do simulations and have the mathematics to show it is correct.

Re: p-Hacking and False Discovery in A/B Testing

#92

Post by rpier001 »

tedsanders wrote: post #52
aalleavitch wrote: post #51

Earlier quoted context omitted.

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward. Completely agree with Evan Miller here, thank you for sharing the link.

Except that the quote reflects a common misunderstanding. The problem of optimal stopping is mostly a function of decision making over multiple looks. A Bayesian approach that makes decisions over multiple looks has similar issues. This can be mitigated by strong priors, but typically to an unknown degree and at some cost to 'power'. How/why does this claim arise? It is because a 'true' Bayesian approach makes no decisions/inferences - it just describes the current state of knowledge. If describing the distribution of the posterior, then 'stop at any time and be valid' else 'you got your Neyman-Pearson in my Bayesian analysis'.

Re: p-Hacking and False Discovery in A/B Testing

#93

Post by YeGoblynQueenne »

tedsanders wrote: post #50
boron1006 wrote: post #20

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…

Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…

>> If an effect is strong, it will show up earlier.

But if an effect shows up earlier, it is not necessarily strong. I think that's the point with running an experiment for a predeterimned length- so you know you didn't get un/lucky and hit a clump of results at the start of the experiments that will sort of average out later on.

Obviously, if a trial drug is killing your patients at a surprising rate, you need to stop the experiment. In fact, I believe experiments are sometimes stopped on ethical grounds when a drugs is found to heal the experiment group at a high rate, also, either so that the control group can also benefit, or just because it is hard to justify giving only half of your patients a life-saving drug and a placebo to the rest.

But those are ethical considerations - not practical ones. Such experiments are cut short without complete confidence to the results, when there is the merest hint of ethical issues down the line. At least that's my understanding.

Re: p-Hacking and False Discovery in A/B Testing

#94

Post by srean »

smallnamespace wrote: post #82
yichijin wrote: post #78

Earlier quoted context omitted.

Hey all, statistician from Optimizely chiming in here. Just wanted to point out that this is exactly the right point. I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple look…

What's the tradeoff vs. just taking a direct Bayesian approach? In fact, why use an inferential framework at all (estimating some sort of probability and using it to guide action), rather than directly using a policy learning framework, e.g. modeling this as Q-learning or multi-armed bandit problem? If at the end of the day you have some objective function (e.g. 'making money'), some known space of actions (e.g. move…

> If at the end of the day you have some objective function

That "If" often evaluates to false.

There are tough judgement calls involved in selecting what is that metric that the org wants to optimize. It is very rare that business management commits to a clear quantitative goal. Reasons are many -- weasel room is important politically, selecting a metric that captures short term and long term goals is difficult, there is a lot of uncertainty in the costs due to uncertainty on how overhead should be billed etc etc.

This is fairly common. Typically, in these situations its the PMs who make the final call. There the goal of the experiment is to glean as much knowledge as possible, and present it to the PM. If that comes at the cost of exposing some customers to bad choices, so be it -- in other words, explore at the cost of losses in the opportunity to exploit.

Re: p-Hacking and False Discovery in A/B Testing

#95

Post by srean »

Jabbles wrote: post #83
yichijin wrote: post #77

Earlier quoted context omitted.

Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as you're not using a method designed to be checked at a single point in time. Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects). The type of test conducted in cl…

Presumably using your method takes longer/requires more samples than a method that only checks once?

I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.

Re: p-Hacking and False Discovery in A/B Testing

#96

Post by carlmr »

glup wrote: post #85
p10_user wrote: post #40

Earlier quoted context omitted.

It’s done in academia too. Along with “remove the outlier data that messes up our p values”

The difference is that the people in academia know that it is statistically unsound and choose to act unethically. I think the problem in A/B testing is that a lot of developers don't know it is unsound.

This is one of the cases where self-taught developers are usually not as good as those with a "proper" education. In uni you'll learn a lot of tangentially related stuff like p-hacking and design of experiment which a lot of people won't pick up when self-taught.

Re: p-Hacking and False Discovery in A/B Testing

#97

Post by whyever »

p10_user wrote: post #40
nanis wrote: post #23

Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…

It’s done in academia too. Along with “remove the outlier data that messes up our p values”

It might make sense to remove systematic outliers if you know they are from non-statistical effects.

Re: p-Hacking and False Discovery in A/B Testing

#98

Post by gbrown »

tedsanders wrote: post #50
boron1006 wrote: post #20

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…

Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…

Adaptive designs are great, but especially in low risk areas like business A/B testing, I'll settle for more people understanding that statistical models are not magic black box truth detectors, and that p-values lose their interpretation in the presence of exploratory practices.

The world would probably be a better place if we taught introductory statistics from a Bayesian perspective, but people get pretty set on their ways.

Re: p-Hacking and False Discovery in A/B Testing

#99

Post by gbrown »

boron1006 wrote: post #55
tedsanders wrote: post #50

Earlier quoted context omitted.

Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…

Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake. In a realistic scenario for us, bayesian and frequentist approaches will probably converge to a point that's close enough for a company that runs a website (i.e., we weren't killing anyone by leaving our experiment running). We also weren't getting massive fluctuations in effect size. The cost o…

Adaptive designs and early stopping rules are really good when:

1. Data collection is expensive (time or money)

2. Keeping with the status quo in the presence of new evidence is problematic (withholding a promising new drug)

3. Continuing with the experiment in the presence of new evidence is problematic (the new drug is hurting people)

Absent one of those features, it's probably not worth the added complexity.

Re: p-Hacking and False Discovery in A/B Testing

#100

Post by geoprofi »

My very recent meta-analysis of 115 A/B tests reveals that a large proportion are highly suspect for p-hacking: http://blog.analytics-toolkit.com/2018/analysis-of-115-a-b-t...

Going the Bayesian way, as suggested in some comments, is no solution at all, as I am not aware of an accepted Bayesian approach to dealing with the issue:

http://blog.analytics-toolkit.com/2017/bayesian-ab-testing-n...

(feel free to run sims, if you do not trust the logic ;-)) as well as on a more general level:

http://blog.analytics-toolkit.com/2017/5-reasons-bayesian-ab...

Post reply on HN