Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

p-Hacking and False Discovery in A/B Testing

papers.ssrn.com

Post reply on HN
Print view View original

61–70 of 106 posts

Re: p-Hacking and False Discovery in A/B Testing

#61

Post by gingerlime »

I created an open source A/B test framework[0], which also uses Bayesian analysis on the dashboard. IANAS(tatistician), but from what I understand it’s still better to plan the check point in advance, rather than stop when reaching significance.

A couple of articles worth reading [1] [2] (can’t exactly vouch for their validity but seem to make some good arguments that appear thought out)

[0] https://github.com/Alephbet/gimel

[1] http://varianceexplained.org/r/bayesian-ab-testing/

[2] http://blog.analytics-toolkit.com/2017/the-bane-of-ab-testin...

Re: p-Hacking and False Discovery in A/B Testing

#62

Post by tzahola »

havkom wrote: post #42
cle wrote: post #19

Traditional A/B testing has very poor ergonomics. Experimenters are usually put in awkward conflict-of-interest situations that create multiple strong incentives not to perform rigorous, disciplined, valid experiments. Null hypothesis significance testing is fundamentally misaligned with business needs and is not a good tool for businesses. This is true in many fields of science as well, but at least they have some m…

Well, I have had well run experiments which showed that the “currently internally hyped” way of doing things is completely inferior to the “old boring” and inexpensive way. The project leader was extremely hyped about doing the experiment to prove the superiority of the new hyped way. When he got the results though, it was clear that this was not to be talked about and this result would not be presented to his superi…

“We’re a data-driven organization! [as long as the data fits our agenda]”

It’s one of my favorite methodologies, next to “agile waterfall” and “holocracy with managers, middle-managers and minibosses”.

Re: p-Hacking and False Discovery in A/B Testing

#63

Post by baybal2 »

Seconding this, the number of e-commerce companies who got "A/B testing into bankruptcy" on my memory approaches 20.

My take on this. Even in cases where such testing was done by disciplined statisticians (which is not the case in at least 9 times out of 10. Yes, a math or cs PhD major is not a professional statistician by any stretch,) the value of advice made from that data is marginal at best.

As eCommerce is bread and butter of cheap electronics industry, I saw times and times again that "science driven" outfits loose out to others. Not so much because of their quality of decision making was demonstrably inferior, but because their obsession with "statistical tasseography" drained their resources, and shifted their focus away from things of obvious importance.

Re: p-Hacking and False Discovery in A/B Testing

#64

Post by baybal2 »

boron1006 wrote: post #20

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…

Well, think of value of any analytics in eCommerce setting:

1. you see a random SKU spiking for n month in a row - good, keep stocking it, maybe even spend few adwords vouchers on it. BUT the fact that nobody in the company readily knows the nature of such spike is already an indication that the business fails at understanding its market.

2. Never in my career, I saw any of "coffee divination" level ideas coming from analysts ever being "life changing" for a company. I been through a number companies spending money on "algorithmic optimisation service" for banner ads of their clients. Yes, the "optimised" banners did score progressively more clicks with time, yet a single full time designer who was hired alongside the "optimiser" company could be scoring more clicks and purchases - without any input from any kind of data analysis

Re: p-Hacking and False Discovery in A/B Testing

#65

Post by smichel17 »

cepth wrote: post #12
nvahalik wrote: post #2

ELI5... what’s a p-hack?

If you’ve taken some statistics or econometrics, you’ve probably heard of “significance levels” and “p-values”. For some reason, academia choose 0.05 as a threshold for “meaningful” or “significant” results. Generally, a 0.05 p-value means that you would observe your result in 5% of experiments due to random sampling error. I.e. if I tested “is X correlated with cancer”, and my null hypothesis is “X isn’t correlated…

The problem with 0.05 isn't how lenient it is, but rather the fact that a default exists at all.

A p-value should be chosen (before running the experiment!) based on how confident the researcher wants to be in their result.

For my high school statistics final project, I did an experiment to test whether a stupid prank/joke was funny. Had a pretty terrible experimental design (tons of bias) and tiny sample size (That would be wildly inappropriate if I were QA testing a new model of airbag or medication. But I wasn't, and I'm not going to use the results for anything other than sharing this anecdote, so it was fine.

Similarly, I'd say in some A/B testing scenarios, it's okay to use a lower standard of proof (though p-hacking is definitely not). Especially if you're just using the test as one piece of information too help you decide on the final design. The problem is when people do bad stats and then use the result as an excuse to throw out their human judgment.

Re: p-Hacking and False Discovery in A/B Testing

#66

Post by tomrod »

baybal2 wrote: post #64
boron1006 wrote: post #20

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…

Well, think of value of any analytics in eCommerce setting: 1. you see a random SKU spiking for n month in a row - good, keep stocking it, maybe even spend few adwords vouchers on it. BUT the fact that nobody in the company readily knows the nature of such spike is already an indication that the business fails at understanding its market. 2. Never in my career, I saw any of "coffee divination" level ideas coming from…

[1] is what I do a lot of discovery on in my own work. The complexity of the digital domain to the sales funnel, as well as the increased reach to new populations, makes simple A/B testing inadequate for accurate causal analysis. I think BigCorp has beneficial scale benefits here.

[2] I've only seen massive product movement when the analysts and idea folks work together. I've never seen an idea wonk hit the bullseye contrary to simple economic theory/reasoning on their customer target.

Re: p-Hacking and False Discovery in A/B Testing

#67

Post by tomrod »

stochastic_monk wrote: post #54
tedsanders wrote: post #52

Earlier quoted context omitted.

>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward. Completely agree with Evan Miller here, thank you for sharing the link.

Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.

Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?

Re: p-Hacking and False Discovery in A/B Testing

#68

Post by dahdum »

raverbashing wrote: post #48

People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't Assuming 'B' is the new option, there are 3 possibilities, A is better than B, A is equivalent to B, A is worse than B If your p-hacked experiment tells you to change from A to B while the null hypothesis was correct, you didn't get much worse off than you were in the first place. And if your long term metrics wer…

A large percentage of experiments I've run were intended to only test if variant was worse than control, we didn't care much how much better the variant may be.

Usually these would be positive consumer facing features we were concerned may negatively affect conversion. The switch to Bayesian made that a lot easier to run.

Re: p-Hacking and False Discovery in A/B Testing

#69

Post by [deleted] »

andreareina wrote: post #46
emodendroket wrote: post #27

Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology

I think that unnecessarily discounts exploratory work. There's nothing wrong with forming a hypothesis after seeing a pattern in data. But remember that it's just a hypothesis -- an unconfirmed guess. After the hypothesis is formulated, then an experiment can be designed to test it, see if its predictions hold up.

[deleted]

Re: p-Hacking and False Discovery in A/B Testing

#70

Post by gwern »

boron1006 wrote: post #21
aisscott wrote: post #15

Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper. If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://w…

I'm curious about the wording "effects are truly null". I was always under the impression that you can never really "accept the null", but rather "fail to reject the alternative".

In your standard NHST test, sure. But you can do different models. If I'm reading OP right, what they do is a mixture model, in which effects are assumed to come either from a zero-mean distribution or a positive-mean distribution with unknown probability _P_/1-_P_ and then you fit the collection of >2k effect sizes to find out what value of _P_ best fits the dataset as a whole. Apparently the best fit assumes that ~70% of effects are actually ~0.

This can also be done nicely with a Bayesian mixture model or a spike-and-slab multilevel model, and that is what is done in "What works in e-commerce - a meta-analysis of 6700 online experiments", Brown & Jones 2017 http://www.qubit.com/sites/default/files/pdf/qubit_meta_anal... (although they don't formulate it in terms of a sharp null but ask the more relevant 'probability of a >0 beneficial effect', which for some kinds of A/B test has a very low prior - like only 15% for 'back to top' A/B tests).

Post reply on HN