[deleted]
A/B test improved your website's conversion rate? Not so fast
21–30 of 68 posts
Re: A/B test improved your website's conversion rate? Not so fast
#22A/B testing can be useful at times but it's largely overrated because you're only discovering the best design out of the ones you test. That means there could be a far better design that you failed include in the experiment. Just because one design converts more than the other doesn't mean it's the design with optimal UX. I've seen many tests where the designs included already had faulty UX. This is why it's better t…
I agree, but that's not the whole story. A/B testing isn't just for identifying the best option, it's for figuring out how much opportunity there is in a particular facet of your business. If you try a few reasonable designs, and some designs have much better performance than others, then it makes sense to continue investing time to further improve. You can take what you've learned and try to come up with even better designs.
On the other hand, if the variants mostly perform the same, why spend more time on it? Go focus elsewhere.
It certainly is a logical possibility that the next design you try will be much more impactful, but after trying several variants unsuccessfully your time is probably better spent elsewhere.
Re: A/B test improved your website's conversion rate? Not so fast
#23A/B tests work fine if the signal you are measuring is strong. This is not the case here. Is it even fine to use the distribution assumptions in the later analysis? Looks like these assumptions combined with a higher conversion rate on day 2 for control is the main reason for the surprising result (control is obviously spread out).
> A/B tests work fine if the signal you are measuring is strong. This is not the case here.
The (fictitious) signal they are discussing here is very strong. Scroll down to the figure labeled "posterior distribution of p" and you can see that the two distributions barely overlap.
Re: A/B test improved your website's conversion rate? Not so fast
#24Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…
You're summary is incorrect. Rather, these are simulated data for a fictitious company. The author is demonstrating a scenario in which a purely frequentist approach to A/B testing can result in erroneous conclusions, whereas a Bayesian approach will avoid that error. The broad conclusions are (as noted explicitly at the end of the article): - The data generating process should dictate the analysis technique(s) - lag…
While I am generally in favor of applying Bayesian approaches, that's overkill for this problem. In their (fictitious) example, the key problem is that they ran their test for too short a time. They already know that the typical lag from visit to conversion on their site is longer than a week, which means that if they want to learn the effect on conversions a week isn't enough data.
While it is possible to make some progress on this issue with careful math, simply running the test longer is a far more effective and robust approach.
Re: A/B test improved your website's conversion rate? Not so fast
#25I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.
I've had a similar experience. Some companies will do things like A/B test fonts and button colors, yet ignore bigger things like content, it's absurd.
Re: A/B test improved your website's conversion rate? Not so fast
#26Every marketing team should have a data scientist and A/B tests are almost always done incorrectly.
Re: A/B test improved your website's conversion rate? Not so fast
#27A/B testing can be useful at times but it's largely overrated because you're only discovering the best design out of the ones you test. That means there could be a far better design that you failed include in the experiment. Just because one design converts more than the other doesn't mean it's the design with optimal UX. I've seen many tests where the designs included already had faulty UX. This is why it's better t…
It’s another hill climbing algorithm, with all of the same problems.
Re: A/B test improved your website's conversion rate? Not so fast
#28I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.
My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.
Before any effort goes into something like this I always raise my hand and ask, "If the data shows us something we don't want to see, will we change our strategy? If not, I'd rather put time/effort into other projects." It works about 80% of the time.
Metrics are only useful if the organization is actually willing to learn lessons from them.
Re: A/B test improved your website's conversion rate? Not so fast
#29Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…
Also it might be hard to ensure you aren't externalising "cost"
Re: A/B test improved your website's conversion rate? Not so fast
#30If a carpenter told me he preferred screws over nails or vice versa I would think he was probably just a bad carpenter.
If he went on to post about his preference on the Internet with some made up examples I would be sure that he couldn't be trusted.