A/B testing mistakes I learned the hard way
newsletter.posthog.com
A/B testing mistakes I learned the hard way
1–10 of 30 posts
Re: A/B testing mistakes I learned the hard way
#2I recall getting into a heated debate with an analyst at my company over the topic of "peeking" (he was right; I was wrong, but it took me several days to finally understand what he was saying.)
The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.
Re: A/B testing mistakes I learned the hard way
#3#2 is a slippery slope if you don't do it properly.
You might look end up looking at lots of different slices of your data, and you might come to the conclusion, "Oh, it looks like France is statistically significant negative on our new signup flow changes".
It's important to make sure you have a hypothesis for the given slice before you start the experiment and not just hunt for outliers after the fact, or otherwise you're just p-hacking [1].
Re: A/B testing mistakes I learned the hard way
#4The article says 'Changing the color of the "Proceed to checkout" button will increase purchases.' is a bad hypothesis because it is underspecified.
But what else is there to measure other than checkout button click count(and follow up purchases) to measure the effect of button color change?
Or perhaps this is not a robust example to illustrates undespeficaition?
Re: A/B testing mistakes I learned the hard way
#5I feel like I've too often seen in products new (anti)features that are way too easy to accidentally click, and whenever I do accidentally click I just imagine it's increasing some statistics counter that's ultimately showing the product managers super high engagement, clearly meaning the users must love it to be using it all the time
Re: A/B testing mistakes I learned the hard way
#6#2 is a slippery slope if you don't do it properly. You might look end up looking at lots of different slices of your data, and you might come to the conclusion, "Oh, it looks like France is statistically significant negative on our new signup flow changes". It's important to make sure you have a hypothesis for the given slice before you start the experiment and not just hunt for outliers after the fact, or otherwise…
[deleted]
Re: A/B testing mistakes I learned the hard way
#7I recall getting into a heated debate with an analyst at my company over the topic of "peeking" (he was right; I was wrong, but it took me several days to finally understand what he was saying.) The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.
This is the most "damned if you do, damned if you don't" part of testing. I've found so many coding errors that weren't obvious until you looked at the day 2 or day 3 test results. "Hm, that's weird. Why is $thing happening in this test? It shouldn't even touch that component."
If you peek, you really have to commit to running the test for the full duration no matter what.
Re: A/B testing mistakes I learned the hard way
#8I recall getting into a heated debate with an analyst at my company over the topic of "peeking" (he was right; I was wrong, but it took me several days to finally understand what he was saying.) The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.
This is the most "damned if you do, damned if you don't" part of testing. I've found so many coding errors that weren't obvious until you looked at the day 2 or day 3 test results. "Hm, that's weird. Why is $thing happening in this test? It shouldn't even touch that component." If you peek, you really have to commit to running the test for the full duration no matter what.
No you don't. If your protocol involves peeking (and early stopping), you need different thresholds to declare statistical significance. But you can do that. You just need to know whether you're peeking or not, which everybody does.
Re: A/B testing mistakes I learned the hard way
#9[deleted]
Re: A/B testing mistakes I learned the hard way
#10That's not Simpson's paradox!
> In fact, while the new flow worked great on mobile, conversion was lower on desktop – an insight we missed when we combined these metrics.
> This phenomenon is known as Simpson's paradox – i.e. when experiments show one outcome when analyzed at an aggregated level, but a different one when analyzed by subgroups.
There's nothing strange about finding out that some groups benefit and others lose out when diving up you data. You're looking at an average and some parts are positive and others are negative. Where's the paradox there?
Simpson's paradox is when more button presses lead to more purchases. But then you look at desktop vs mobile and you find out that for both desktop and mobile more clicks doesn't mean more purchases (or worse, more clicks means fewer purchases).
That's why it's a paradox. The association between two variables exists at the aggregate level but doesn't exist or is backwards when you split up the population. It's not a statement about the average performance of something.
I would add a 7th A/B testing mistake to that list and it's not learning about basic probability, statical tests, power, etc. Flying by the seat of your pants when statistics are involved always ends badly.