Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing mistakes I learned the hard way

newsletter.posthog.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: A/B testing mistakes I learned the hard way

#21

Post by aflag »

jakevoytko wrote: post #7
sokoloff wrote: post #2

I recall getting into a heated debate with an analyst at my company over the topic of "peeking" (he was right; I was wrong, but it took me several days to finally understand what he was saying.) The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.

This is the most "damned if you do, damned if you don't" part of testing. I've found so many coding errors that weren't obvious until you looked at the day 2 or day 3 test results. "Hm, that's weird. Why is $thing happening in this test? It shouldn't even touch that component." If you peek, you really have to commit to running the test for the full duration no matter what.

If there is a bug, then the experiment needs to be called off and a new one constructed. You shouldn't change anything else during the execution of the experiment.

Re: A/B testing mistakes I learned the hard way

#22

Post by eterm »

light_hue_1 wrote: post #10

That's not Simpson's paradox! > In fact, while the new flow worked great on mobile, conversion was lower on desktop – an insight we missed when we combined these metrics. > This phenomenon is known as Simpson's paradox – i.e. when experiments show one outcome when analyzed at an aggregated level, but a different one when analyzed by subgroups. There's nothing strange about finding out that some groups benefit and oth…

Yep, I came here to say the same thing. The author has misremembered / misrepresented Simpson's paradox, which is much stronger than an aggregate hiding a group effect.

Re: A/B testing mistakes I learned the hard way

#24

Post by Dyac »

clarle wrote: post #3

#2 is a slippery slope if you don't do it properly. You might look end up looking at lots of different slices of your data, and you might come to the conclusion, "Oh, it looks like France is statistically significant negative on our new signup flow changes". It's important to make sure you have a hypothesis for the given slice before you start the experiment and not just hunt for outliers after the fact, or otherwise…

Fundamentally, you can't use the same data to both generate and validate/disprove a hypothesis.

Srgmenting and data dredging is fine provided you run a new test with fresh data to validate if there is a causal relationship in any correlations found.

Re: A/B testing mistakes I learned the hard way

#25

Post by ImageXav »

Here's another one that I feel is often overlooked by traditional A/B testers: if you have multiple changes, don't simply test them independently. Learn about fractional factorial experiments and interactions, and design your experiment accordingly. You'll get a much more relevant result.

My impression is that companies like to add/test a lot of features separately - and individually these features are good, but together they form complex clutter and end up being a net negative.

Re: A/B testing mistakes I learned the hard way

#26

Post by beejiu »

jakevoytko wrote: post #7
sokoloff wrote: post #2

I recall getting into a heated debate with an analyst at my company over the topic of "peeking" (he was right; I was wrong, but it took me several days to finally understand what he was saying.) The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.

This is the most "damned if you do, damned if you don't" part of testing. I've found so many coding errors that weren't obvious until you looked at the day 2 or day 3 test results. "Hm, that's weird. Why is $thing happening in this test? It shouldn't even touch that component." If you peek, you really have to commit to running the test for the full duration no matter what.

> If you peek, you really have to commit to running the test for the full duration no matter what.

It's more complicated, but you can also run sequential A/B testings using [SPRT](https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...) or similar, where a test gets accepted or rejected once it hits a threshold. I won't go into the details, but you can incrementally calculate the test statistic, so if your test is performing very badly or well, the test will end early.

One product team I worked in run all tests as sequential tests. If you build a framework around this, I'd argue it's easier for statistics-unaware stakeholders to understand when you _can_ end a test early.

Re: A/B testing mistakes I learned the hard way

#27

Post by Jemaclus »

taneq wrote: post #16
Jemaclus wrote: post #13

Earlier quoted context omitted.

To your point, my company is doing some A/B tests and I insisted that we not just measure conversion ("it works") and additionally measure some metrics that would indicate that it works _well_. For example, if you have a carousel of products, and someone buys something from the carousel, then it "works," but it would work _better_ if the item they bought was the first thing on the carousel rather than the last. That…

> but it would work _better_ if the item they bought was the first thing on the carousel rather than the last Depends, does that increase overall sales? Or is it ‘better’ to make the customer ‘walk past’ the other items to get to the thing they want (the way supermarkets make you walk up the back of the shop to get to the milk), and maybe buy something else too?

Absolutely! It's important to have a hypothesis and test it, to OP's point!

Re: A/B testing mistakes I learned the hard way

#28

Post by gregbarbosa »

iamcreasy wrote: post #4

The article says 'Changing the color of the "Proceed to checkout" button will increase purchases.' is a bad hypothesis because it is underspecified. But what else is there to measure other than checkout button click count(and follow up purchases) to measure the effect of button color change? Or perhaps this is not a robust example to illustrates undespeficaition?

> But what else is there to measure other than checkout button click count(and follow up purchases) to measure the effect of button color change?

Purchases occur on the checkout page itself. Its design, payment input, and upsells can all impact results, potentially counteracting the button color's effects. You need a clearer hypothesis to address these.

Re: A/B testing mistakes I learned the hard way

#29

Post by gregbarbosa »

light_hue_1 wrote: post #10

That's not Simpson's paradox! > In fact, while the new flow worked great on mobile, conversion was lower on desktop – an insight we missed when we combined these metrics. > This phenomenon is known as Simpson's paradox – i.e. when experiments show one outcome when analyzed at an aggregated level, but a different one when analyzed by subgroups. There's nothing strange about finding out that some groups benefit and oth…

> Simpson's paradox is when more button presses lead to more purchases. But then you look at desktop vs mobile and you find out that for both desktop and mobile more clicks doesn't mean more purchases (or worse, more clicks means fewer purchases).

How could more button presses lead to increased conversion rates while hiding this data when comparing desktop and mobile? Wouldn’t you see at least one device type demonstrating higher CVR to reflect aggregate CVR increase?

Re: A/B testing mistakes I learned the hard way

#30

Post by light_hue_1 »

gregbarbosa wrote: post #29
light_hue_1 wrote: post #10

That's not Simpson's paradox! > In fact, while the new flow worked great on mobile, conversion was lower on desktop – an insight we missed when we combined these metrics. > This phenomenon is known as Simpson's paradox – i.e. when experiments show one outcome when analyzed at an aggregated level, but a different one when analyzed by subgroups. There's nothing strange about finding out that some groups benefit and oth…

> Simpson's paradox is when more button presses lead to more purchases. But then you look at desktop vs mobile and you find out that for both desktop and mobile more clicks doesn't mean more purchases (or worse, more clicks means fewer purchases). How could more button presses lead to increased conversion rates while hiding this data when comparing desktop and mobile? Wouldn’t you see at least one device type demonst…

That's Simpson's paradox!

You can take data where as a whole presses lead to more purchases. Then split it into two halves (like mobile vs desktop) and show that on both halves presses lead to fewer purchases.

The whole paradox is that the intuition we have for averages doesn't apply to correlations.

I suggest checking out the Wikipedia page.

Post reply on HN