I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…
p-Hacking and False Discovery in A/B Testing
51–60 of 106 posts
Re: p-Hacking and False Discovery in A/B Testing
#52Earlier quoted context omitted.
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…
http://www.evanmiller.org/how-not-to-run-an-ab-test.html
>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward.
Completely agree with Evan Miller here, thank you for sharing the link.
Re: p-Hacking and False Discovery in A/B Testing
#53How can A/B testing tools be improved to prevent p-value hacking? Could it be as simple as declaring your test duration before starting the experiment, and having the tool add an asterisk to your results if you stop the experiment early?
- You must choose the sample size BEFORE
- You must choose the significance level BEFORE (0.05 is not always the best choice)
Re: p-Hacking and False Discovery in A/B Testing
#54Earlier quoted context omitted.
http://www.evanmiller.org/how-not-to-run-an-ab-test.html
>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward. Completely agree with Evan Miller here, thank you for sharing the link.
Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
Re: p-Hacking and False Discovery in A/B Testing
#55I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…
Agree that bayesian approaches would be better here, but disagree that running experiments for preset lengths is a mistake.
In a realistic scenario for us, bayesian and frequentist approaches will probably converge to a point that's close enough for a company that runs a website (i.e., we weren't killing anyone by leaving our experiment running). We also weren't getting massive fluctuations in effect size.
The cost of the Bayesian approach, in terms of learning a new system of statistics, programming everything up from scratch, and interpreting results, probably wouldn't be worth the efficiency gains. If we were creating an AB testing program, then I would probably do so.
Re: p-Hacking and False Discovery in A/B Testing
#56How can A/B testing tools be improved to prevent p-value hacking? Could it be as simple as declaring your test duration before starting the experiment, and having the tool add an asterisk to your results if you stop the experiment early?
Optimizely claim to have since fixed this. The paper where they introduce the method is here: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
Re: p-Hacking and False Discovery in A/B Testing
#57People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't Assuming 'B' is the new option, there are 3 possibilities, A is better than B, A is equivalent to B, A is worse than B If your p-hacked experiment tells you to change from A to B while the null hypothesis was correct, you didn't get much worse off than you were in the first place. And if your long term metrics wer…
Yes not to mention that for nearly all A/B tests the size of the effect is minimal anyway. I have found that if the effect is large you don't need statistics and if it is small it doesn't matter.
Re: p-Hacking and False Discovery in A/B Testing
#58Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
It's perfectly fine to run an experiment until you get a significant result. You just have to do the maths differently - that's what most people don't know.
Re: p-Hacking and False Discovery in A/B Testing
#59Re: p-Hacking and False Discovery in A/B Testing
#60I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…
It's no fun being labelled as 'fussy'. As a researcher I often work with doctors who ask for analysis with p-values on all sorts of inappropriate data sets, pushing back and telling them "you cannot draw any valid inference from this" can be quite hard.