Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B test improved your website's conversion rate? Not so fast

blog.alexandervolkmann.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: A/B test improved your website's conversion rate? Not so fast

#31

Post by kqr »

xibalba wrote: post #7
jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

You're summary is incorrect. Rather, these are simulated data for a fictitious company. The author is demonstrating a scenario in which a purely frequentist approach to A/B testing can result in erroneous conclusions, whereas a Bayesian approach will avoid that error. The broad conclusions are (as noted explicitly at the end of the article): - The data generating process should dictate the analysis technique(s) - lag…

> - The data generating process should dictate the analysis technique(s)

And to expand on this, the data generating process is not about a statistical distribution or any other theoretical construct. Only in the frequentist world do you start with assuming a generating process (for the null hypothesis, specifically).

The data generating process in this case are living, breathing humans doing things humans do.

Re: A/B test improved your website's conversion rate? Not so fast

#32

Post by dredmorbius »

SwiftyBug wrote: post #12
marban wrote: post #3

It sucks to suck at maths.

I'm not sure why you are being downvoted. I completely agree with you: it sucks to suck at maths because I believe this article brings very interesting information, information that could be very useful to me, that I simply cannot understand because I suck at maths.

The commment is vague, uncler, reads as a shallow dismissal, and doesn't inform.

It may be valid. It's insufficient as it stands to make a determination.

TL;DR: it's noise.

Re: A/B test improved your website's conversion rate? Not so fast

#33

Post by kqr »

sosborn wrote: post #28
nrjames wrote: post #10

Earlier quoted context omitted.

My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.

Before any effort goes into something like this I always raise my hand and ask, "If the data shows us something we don't want to see, will we change our strategy? If not, I'd rather put time/effort into other projects." It works about 80% of the time. Metrics are only useful if the organization is actually willing to learn lessons from them.

Oh yeah, defining the thresholds and their associated courses of action ahead of time is important to make a good decision.

Any time someone wants to measure something, the top two questions should be "what are the lower and upper bounds this value has to exceed for us to do something different?"

Very often, it turns out these thresholds for change are so astronomical that nobody thinks we have even the slightest chance of exceeding them. That means the measurement is completely useless. Whatever result we plausibly get, it won't change anything.

You can save a lot of time this way!

Re: A/B test improved your website's conversion rate? Not so fast

#34

Post by OJFord »

jefftk wrote: post #24
xibalba wrote: post #7

Earlier quoted context omitted.

You're summary is incorrect. Rather, these are simulated data for a fictitious company. The author is demonstrating a scenario in which a purely frequentist approach to A/B testing can result in erroneous conclusions, whereas a Bayesian approach will avoid that error. The broad conclusions are (as noted explicitly at the end of the article): - The data generating process should dictate the analysis technique(s) - lag…

While I am generally in favor of applying Bayesian approaches, that's overkill for this problem. In their (fictitious) example, the key problem is that they ran their test for too short a time. They already know that the typical lag from visit to conversion on their site is longer than a week, which means that if they want to learn the effect on conversions a week isn't enough data. While it is possible to make some…

I'm no statistician, but don't you have the same problem however long you run it? Giving even more time for slow conversions to amass?

Also you and GP are calling the example fictitious, but seems to based on 'real traffic logs' via https://dl.acm.org/doi/10.1145/2623330.2623634

Re: A/B test improved your website's conversion rate? Not so fast

#35

Post by ssharp »

lewisl9029 wrote: post #15
codeptualize wrote: post #11

Earlier quoted context omitted.

It seems that any kind of data is mainly used to confirm the believes people already had and/or justify decisions already made.

Not to mention the decision paralysis and change aversion it often introduces into company culture where every change, however trivial or however obviously beneficial, has to first go through a 2-week A/B test which often turns out to be inconclusive anyways, and sometimes takes more eng resources to set up and run than it takes to make the change itself.

> however trivial

Previous testing should give the company at least some baseline understanding of what is trivial and what isn't. The correct way to experiment is certainly not "let's experiment every idea!"

> however obviously beneficial

If you've been around long enough, you've almost certainly run into dozens of "obviously beneficial" changes that led to poorer performance.

Most of what you're describing is issues with poor prioritization, a lack of understanding about your audience, and a culture that has a difficult time making decisions.

Re: A/B test improved your website's conversion rate? Not so fast

#36

Post by ssharp »

antux wrote: post #18

A/B testing can be useful at times but it's largely overrated because you're only discovering the best design out of the ones you test. That means there could be a far better design that you failed include in the experiment. Just because one design converts more than the other doesn't mean it's the design with optimal UX. I've seen many tests where the designs included already had faulty UX. This is why it's better t…

I'm not entirely sure what the point here is. You're calling A/B testing overrated because it doesn't involve UX designers? I'd agree that having some UX resources in test design is critical, so if that's being done up to standard, is testing still overrated?

Re: A/B test improved your website's conversion rate? Not so fast

#37

Post by ssharp »

throw1234651234 wrote: post #8

Google Optimize flickers the screen with React. And makes non-mobile layouts appear mobile (messes with screen width presets). That is all.

There is almost certainly a workaround with the flicker issue.

I'm not sure I've ever encountered the second issue with screen width.

Re: A/B test improved your website's conversion rate? Not so fast

#38

Post by vlozko »

This article reminded me of an experience I had as a developer for an online retailer. The product browse team, the one responsible for showing lists of products from searches, categories, brands, etc., had a slew of AB tests to see product detail page viewing conversion rate. One of these tests included the bright idea of removing the product name from the individual product cards. Trouble is, the names often contained differentiating descriptors. While dogfooding the app, I was thoroughly frustrated with the need to constantly go back and forth between browse and detail pages. When I spoke to the browse team about it, they were patting themselves on the back on how amazing their detail page viewing conversation rates were. Made me a skeptic of AB tests since.

Re: A/B test improved your website's conversion rate? Not so fast

#39

Post by xibalba »

OJFord wrote: post #34
jefftk wrote: post #24

Earlier quoted context omitted.

While I am generally in favor of applying Bayesian approaches, that's overkill for this problem. In their (fictitious) example, the key problem is that they ran their test for too short a time. They already know that the typical lag from visit to conversion on their site is longer than a week, which means that if they want to learn the effect on conversions a week isn't enough data. While it is possible to make some…

I'm no statistician, but don't you have the same problem however long you run it? Giving even more time for slow conversions to amass? Also you and GP are calling the example fictitious, but seems to based on 'real traffic logs' via https://dl.acm.org/doi/10.1145/2623330.2623634

We're taking the author at his word:

> "Let us consider the following fictitious example in which Larry the analyst of the internet company Nozama"

Nozama is Amazon backwards.

Re: A/B test improved your website's conversion rate? Not so fast

#40

Post by temple__ »

ssharp wrote: post #37
throw1234651234 wrote: post #8

Google Optimize flickers the screen with React. And makes non-mobile layouts appear mobile (messes with screen width presets). That is all.

There is almost certainly a workaround with the flicker issue. I'm not sure I've ever encountered the second issue with screen width.

https://support.google.com/optimize/answer/7100284?hl=en
Post reply on HN