A/B test improved your website's conversion rate? Not so fast • Hacker News Board

Re: A/B test improved your website's conversion rate? Not so fast

#41

Post by ssharp » Fri, Jan 07, 2022, 4:57 PM UTC

This whole example seems like it boils down to a poor test/analysis plan than anything that truly speaks to the value of Bayesian approaches:

1) It's almost always a bad idea to decide a test based on one-week's worth of data, regardless of what statistical approach you take

2) There's not really any info on why Fisher's exact test is used. It seems like most A/B software has adopted Bayesian but the ones who haven't, I believe, choose the Student's test and require prior sample sizing

3) The conversion delay issue was not addressed in the measurement plan. There are clear ways to address this issue, both tactically as well as mathematically. From a tactical standpoint, most testing platforms, you'd be able to change the test allocation to 0%, which would allowed previously bucketed users to continue to be measured with subsequent visits while not allowing any new users in. You could also just run the test long enough to where the conversion lag no longer has a major impact on results (this may or may not be possible, depending on how long and fat the lag tail is).

Re: A/B test improved your website's conversion rate? Not so fast

#42

Post by knuthsat » Fri, Jan 07, 2022, 4:59 PM UTC

jefftk wrote: post #23

knuthsat wrote: post #17

A/B tests work fine if the signal you are measuring is strong. This is not the case here. Is it even fine to use the distribution assumptions in the later analysis? Looks like these assumptions combined with a higher conversion rate on day 2 for control is the main reason for the surprising result (control is obviously spread out).

> A/B tests work fine if the signal you are measuring is strong. This is not the case here. The (fictitious) signal they are discussing here is very strong. Scroll down to the figure labeled "posterior distribution of p" and you can see that the two distributions barely overlap.

Yes, I saw the figure and that's why I commented that the day 2 conversions for the control are basically giving all of the information in the assumed model.

To me it just looks like a whole new batch of assumptions. Might be fictitiously valid or not.

Re: A/B test improved your website's conversion rate? Not so fast

#43

Post by ssharp » Fri, Jan 07, 2022, 5:00 PM UTC

temple__ wrote: post #40

ssharp wrote: post #37

Earlier quoted context omitted.

There is almost certainly a workaround with the flicker issue. I'm not sure I've ever encountered the second issue with screen width.

https://support.google.com/optimize/answer/7100284?hl=en

Anti-flicker snippet is definitely the first step. Since this is a SPA and the flicker may be caused well past the point of the initial pageview, there may also be an issue with how the code is written and hooked into the SPA framework, in this case React.

Re: A/B test improved your website's conversion rate? Not so fast

#44

Post by omegalulw » Fri, Jan 07, 2022, 5:28 PM UTC

jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

Wouldn't it be better to run the experiment longer _and_ discard the data from initial few weeks?

Re: A/B test improved your website's conversion rate? Not so fast

#45

Post by jspash » Fri, Jan 07, 2022, 5:43 PM UTC

snarfy wrote: post #9

I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.

We stopped doing A/B tests after I insisted that they all be done as A/A/B tests. Suddenly the "clear winners" weren't so clear after all. It confused and frustrated the marketing department so much that it was decided just to stop doing them all together.

The reason I wanted this type of test was because it was a waste of time testing shades of blue or two headlines that only differed by 2 words. The test variants were never radical enough to see any kind of significant uplift. Then after 5-10 tests the design starts to suffer by wandering down some weird path that nobody would consciously design from the outset. But the series of test "winners" made things go off in wild directions.

I still think there is some value in A/B testing (A/A/B only, if I'm honest). But in a small team, it's a waste of time.

Re: A/B test improved your website's conversion rate? Not so fast

#46

Post by pottertheotter » Fri, Jan 07, 2022, 6:06 PM UTC

jspash wrote: post #45

snarfy wrote: post #9

I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.

We stopped doing A/B tests after I insisted that they all be done as A/A/B tests. Suddenly the "clear winners" weren't so clear after all. It confused and frustrated the marketing department so much that it was decided just to stop doing them all together. The reason I wanted this type of test was because it was a waste of time testing shades of blue or two headlines that only differed by 2 words. The test variants w…

For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?

Re: A/B test improved your website's conversion rate? Not so fast

#47

Post by rwilson4 » Fri, Jan 07, 2022, 6:13 PM UTC

kqr wrote: post #31

xibalba wrote: post #7

Earlier quoted context omitted.

You're summary is incorrect. Rather, these are simulated data for a fictitious company. The author is demonstrating a scenario in which a purely frequentist approach to A/B testing can result in erroneous conclusions, whereas a Bayesian approach will avoid that error. The broad conclusions are (as noted explicitly at the end of the article): - The data generating process should dictate the analysis technique(s) - lag…

> - The data generating process should dictate the analysis technique(s) And to expand on this, the data generating process is not about a statistical distribution or any other theoretical construct. Only in the frequentist world do you start with assuming a generating process (for the null hypothesis, specifically). The data generating process in this case are living, breathing humans doing things humans do.

The data generating process is the random assignment of people to experiment groups.

The potential outcomes are fixed: if a person is assigned to one group the outcome is x1; if another, x2. No assumption is made about these potential outcomes. They are not considered random, unless the Population Average Treatment Effect is being estimated. And even in that case, no distribution is assumed. It certainly is not Gaussian for example.

Under random assignment, the observed treatment effect is unbiased for the Sample Average Treatment Effect. So again, the data generating process of interest to the analyst is random assignment.

Re: A/B test improved your website's conversion rate? Not so fast

#48

Post by kqr » Fri, Jan 07, 2022, 6:20 PM UTC

rwilson4 wrote: post #47

kqr wrote: post #31

Earlier quoted context omitted.

> - The data generating process should dictate the analysis technique(s) And to expand on this, the data generating process is not about a statistical distribution or any other theoretical construct. Only in the frequentist world do you start with assuming a generating process (for the null hypothesis, specifically). The data generating process in this case are living, breathing humans doing things humans do.

The data generating process is the random assignment of people to experiment groups. The potential outcomes are fixed: if a person is assigned to one group the outcome is x1; if another, x2. No assumption is made about these potential outcomes. They are not considered random, unless the Population Average Treatment Effect is being estimated. And even in that case, no distribution is assumed. It certainly is not Gauss…

Assuming you're able to actually achieve truly random participation in the various arms you're trialing, you're right.

And it's my fault for not thinking of that as a possibility. Colour me jaded after experiencing very many bad attempts at randomization that actually suffer from Simpson's paradox in various ways!

Re: A/B test improved your website's conversion rate? Not so fast

#49

Post by gpt5 » Fri, Jan 07, 2022, 6:32 PM UTC

omegalulw wrote: post #44

jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

Wouldn't it be better to run the experiment longer _and_ discard the data from initial few weeks?

This could make the entire org/company run and innovate much slower. Ideally you can build better models that predict long term conversion from short term data. These models can be refined with long term experiments.

Re: A/B test improved your website's conversion rate? Not so fast

#50

Post by rwilson4 » Fri, Jan 07, 2022, 6:34 PM UTC

kqr wrote: post #48

rwilson4 wrote: post #47

Earlier quoted context omitted.

The data generating process is the random assignment of people to experiment groups. The potential outcomes are fixed: if a person is assigned to one group the outcome is x1; if another, x2. No assumption is made about these potential outcomes. They are not considered random, unless the Population Average Treatment Effect is being estimated. And even in that case, no distribution is assumed. It certainly is not Gauss…

Assuming you're able to actually achieve truly random participation in the various arms you're trialing, you're right. And it's my fault for not thinking of that as a possibility. Colour me jaded after experiencing very many bad attempts at randomization that actually suffer from Simpson's paradox in various ways!

You're absolutely correct, proper A/B testing has many engineering challenges!