A/B test improved your website's conversion rate? Not so fast
blog.alexandervolkmann.com
A/B test improved your website's conversion rate? Not so fast
1–10 of 68 posts
Re: A/B test improved your website's conversion rate? Not so fast
#2Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group.
You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due to novelty, but then it wears off over time. Running experiments longer helps a lot in both cases.
Re: A/B test improved your website's conversion rate? Not so fast
#3It sucks to suck at maths.
Re: A/B test improved your website's conversion rate? Not so fast
#4Question to HN folks:
What are, in your opinion, the best resources for a Computer Science graduate to learn how to apply statistics like this?
Re: A/B test improved your website's conversion rate? Not so fast
#5Question to HN folks: What are, in your opinion, the best resources for a Computer Science graduate to learn how to apply statistics like this?
Bayesian Methods For Hackers is a very popular one.
[0] https://github.com/CamDavidsonPilon/Probabilistic-Programmin...
Re: A/B test improved your website's conversion rate? Not so fast
#6Question to HN folks: What are, in your opinion, the best resources for a Computer Science graduate to learn how to apply statistics like this?
I can highly recommend Richard McElreath's lectures "Statistical Rethinking":
https://www.youtube.com/channel/UCNJK6_DZvcMqNSzQdEkzvzA/pla...
They are based on his text book with the same title.
(Disclaimer: I'm the author of the blog post.)
Re: A/B test improved your website's conversion rate? Not so fast
#7Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…
You're summary is incorrect.
Rather, these are simulated data for a fictitious company. The author is demonstrating a scenario in which a purely frequentist approach to A/B testing can result in erroneous conclusions, whereas a Bayesian approach will avoid that error. The broad conclusions are (as noted explicitly at the end of the article):
- The data generating process should dictate the analysis technique(s)
- lagged response variables require special handling
- Stan propaganda ;) but also :(
It would be cool to understand what the weaknesses or risks of erroneous conclusion to the Bayseian approach in this or similar scenarios. In other words, is it truly a risk-free trade off to switch from a frequentist technique to a Bayesian technique, or are we simply swapping one set of risks for another?
tl;dr The author's point is not to make a general claim about the aggressiveness of CTAs.
Re: A/B test improved your website's conversion rate? Not so fast
#8Google Optimize flickers the screen with React. And makes non-mobile layouts appear mobile (messes with screen width presets). That is all.
Re: A/B test improved your website's conversion rate? Not so fast
#9I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.
Re: A/B test improved your website's conversion rate? Not so fast
#10I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.
My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.