Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B test improved your website's conversion rate? Not so fast

blog.alexandervolkmann.com

Post reply on HN
Print view View original

1–10 of 68 posts

Re: A/B test improved your website's conversion rate? Not so fast

#2

Post by jefftk »

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group.

You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due to novelty, but then it wears off over time. Running experiments longer helps a lot in both cases.

Re: A/B test improved your website's conversion rate? Not so fast

#5

Post by perpetualpatzer »

Darge wrote: post #4

Question to HN folks: What are, in your opinion, the best resources for a Computer Science graduate to learn how to apply statistics like this?

Bayesian Methods For Hackers is a very popular one.

[0] https://github.com/CamDavidsonPilon/Probabilistic-Programmin...

Re: A/B test improved your website's conversion rate? Not so fast

#6

Post by volkale »

Darge wrote: post #4

Question to HN folks: What are, in your opinion, the best resources for a Computer Science graduate to learn how to apply statistics like this?

I can highly recommend Richard McElreath's lectures "Statistical Rethinking": https://www.youtube.com/channel/UCNJK6_DZvcMqNSzQdEkzvzA/pla...

They are based on his text book with the same title.

(Disclaimer: I'm the author of the blog post.)

Re: A/B test improved your website's conversion rate? Not so fast

#7

Post by xibalba »

jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

You're summary is incorrect.

Rather, these are simulated data for a fictitious company. The author is demonstrating a scenario in which a purely frequentist approach to A/B testing can result in erroneous conclusions, whereas a Bayesian approach will avoid that error. The broad conclusions are (as noted explicitly at the end of the article):

- The data generating process should dictate the analysis technique(s)

- lagged response variables require special handling

- Stan propaganda ;) but also :(

It would be cool to understand what the weaknesses or risks of erroneous conclusion to the Bayseian approach in this or similar scenarios. In other words, is it truly a risk-free trade off to switch from a frequentist technique to a Bayesian technique, or are we simply swapping one set of risks for another?

tl;dr The author's point is not to make a general claim about the aggressiveness of CTAs.

Re: A/B test improved your website's conversion rate? Not so fast

#10

Post by nrjames »

snarfy wrote: post #9

I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.

My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.
Post reply on HN