p-Hacking and False Discovery in A/B Testing • Hacker News Board

Re: p-Hacking and False Discovery in A/B Testing

#41

Post by joshuamorton » Sun, Jul 22, 2018, 6:40 AM UTC

cepth wrote: post #36

joshuamorton wrote: post #34

Earlier quoted context omitted.

Not the above poster, but... >I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way Correct, but the most common methods of p-hacking involve changing the dataset size, either by repeating the experiment until the desired result is achieved (a la xkcd [0]), or by removing a large part of the dataset due to a seemingly-legitimate excuse (like the f…

Sorry if the second question was unclear. My point was that for simulation based research, it doesn’t seem that pre-announcing your sample size would do much for preventing p-hacking. E.g. if I say “I will do 10000 runs of my simulation”, what’s to prevent me from doing those runs multiple times, and selecting the one that gives me the desired p-value? For observational research, there’s obviously a physical limit to…

I believe that's where the original post's

>given enough time.

comes in.

One nice thing about simulation based research is that it is often (more) reproducible, so a simulation can be run 10000 times, but then the paper might be expected to report how often the simulation succeeded. In other words, you can increase the simulation size to make p-hacking infeasible

Note that in practice, pre-announcing your sample size doesn't prevent p-hacking unless your sample size is == to a known sample. If you say "our sample size will be X", but you can collect 2 or 3x X data even, you can almost certainly p-hack.

Not to mention that I'm unaware of any field where people actually pre-announce their sample sizes. Does this happen on professor's web pages and I'm unaware, or as footnotes in prior papers?

Re: p-Hacking and False Discovery in A/B Testing

#42

Post by havkom » Sun, Jul 22, 2018, 6:55 AM UTC

cle wrote: post #19

Traditional A/B testing has very poor ergonomics. Experimenters are usually put in awkward conflict-of-interest situations that create multiple strong incentives not to perform rigorous, disciplined, valid experiments. Null hypothesis significance testing is fundamentally misaligned with business needs and is not a good tool for businesses. This is true in many fields of science as well, but at least they have some m…

Well, I have had well run experiments which showed that the “currently internally hyped” way of doing things is completely inferior to the “old boring” and inexpensive way. The project leader was extremely hyped about doing the experiment to prove the superiority of the new hyped way. When he got the results though, it was clear that this was not to be talked about and this result would not be presented to his superiors.

Re: p-Hacking and False Discovery in A/B Testing

#43

Post by Engineering-MD » Sun, Jul 22, 2018, 6:56 AM UTC

emodendroket wrote: post #27

Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology

I would say it is. By not being hypothesis driven, you are producing too many degrees of freedom by de facto testing every comparison.

This is a huge problem in science too. I have regularly been told to just see what happens and come up with hypotheses after, or others have been unable to say what their hypotheses actually are.

Scientists seem less and less likely to be trained in statistics, and in the scientific process. Technical knowledge is important, but understanding of the scientific process is much more important.

Re: p-Hacking and False Discovery in A/B Testing

#44

Post by cepth » Sun, Jul 22, 2018, 7:05 AM UTC

joshuamorton wrote: post #41

cepth wrote: post #36

Earlier quoted context omitted.

Sorry if the second question was unclear. My point was that for simulation based research, it doesn’t seem that pre-announcing your sample size would do much for preventing p-hacking. E.g. if I say “I will do 10000 runs of my simulation”, what’s to prevent me from doing those runs multiple times, and selecting the one that gives me the desired p-value? For observational research, there’s obviously a physical limit to…

I believe that's where the original post's >given enough time. comes in. One nice thing about simulation based research is that it is often (more) reproducible, so a simulation can be run 10000 times, but then the paper might be expected to report how often the simulation succeeded. In other words, you can increase the simulation size to make p-hacking infeasible Note that in practice, pre-announcing your sample size…

Again, academia/research is not my profession. But, some cool efforts in this area include osf.io, which is trying to be the Arxiv or Github of preregistration for scientific studies.

The best preregistration plans will typically include a declared sample or population to observe (http://datacolada.org/64), or at least clear cut criteria for which participants or observations you will exclude.

I think for the type of economics/finance research I’m most familiar with, you often implicitly announce your sample when securing funding for a research proposal. E.g. if I’m trying to see if pursuing a momentum strategy with S&P 500 stocks is profitable (a la AQR’s work), it’s pretty obvious what the sample ought to be. This is partly why that meta study I linked to earlier was able to sniff out potential signs of p-hacking.

Re: p-Hacking and False Discovery in A/B Testing

#45

Post by tzahola » Sun, Jul 22, 2018, 7:15 AM UTC

t3scrote wrote: post #38

raphaelrk wrote: post #8

Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?

We set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers i…

So one in ten of your findings is bogus.

Re: p-Hacking and False Discovery in A/B Testing

#46

Post by andreareina » Sun, Jul 22, 2018, 7:29 AM UTC

emodendroket wrote: post #27

Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology

I think that unnecessarily discounts exploratory work. There's nothing wrong with forming a hypothesis after seeing a pattern in data. But remember that it's just a hypothesis -- an unconfirmed guess. After the hypothesis is formulated, then an experiment can be designed to test it, see if its predictions hold up.

Re: p-Hacking and False Discovery in A/B Testing

#47

Post by pimmen » Sun, Jul 22, 2018, 7:47 AM UTC

boron1006 wrote: post #20

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…

I've seen that culture too. I don't want to throw my colleagues under the bus but the other teams at my company that have done A/B testing to compare design choices don't perform proper controls or pre-register hypothesis. Eg. when they want to divide the two groups, they often divide them based on some classification (gender, age, geography etc.) rather than dividing them up completely randomly.

In my team we try to be methodical. I'm just a lowly engineer but one of my team mates is a statistician and another is a PhD student. We know we need to pre-register the hypothesis and at what significance we're testing for first, divide the groups randomly and run the experiment for a set time period.

We've gotten a lot more negative results and been proven wrong in our guesses than the other teams. For some reason I take pride in that.

Re: p-Hacking and False Discovery in A/B Testing

#48

Post by raverbashing » Sun, Jul 22, 2018, 7:51 AM UTC

People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't

Assuming 'B' is the new option, there are 3 possibilities, A is better than B, A is equivalent to B, A is worse than B

If your p-hacked experiment tells you to change from A to B while the null hypothesis was correct, you didn't get much worse off than you were in the first place. And if your long term metrics were in place then you can get a better measure for your experiment.

Not to mention experimental failures by unaccounted variables

Re: p-Hacking and False Discovery in A/B Testing

#49

Post by t3scrote » Sun, Jul 22, 2018, 7:53 AM UTC

tzahola wrote: post #45

t3scrote wrote: post #38

Earlier quoted context omitted.

We set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers i…

So one in ten of your findings is bogus.

But isn’t that better than blindly introducing changes without testing them at all?

Re: p-Hacking and False Discovery in A/B Testing

#50

Post by tedsanders » Sun, Jul 22, 2018, 8:02 AM UTC

boron1006 wrote: post #20

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…

Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to treat 1,000 patients, thinking that the effect would be small. This is one reason I think Bayesian approaches are better than frequentist approaches for A/B testing.