11–20 of 68 posts
I've dealt with this enough that at this point I'm convinced all companies that do this fail to see the users through the metrics. A/B testing is overvalued.
My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.
It sucks to suck at maths.
It sucks to suck at maths.
I'm not sure why you are being downvoted. I completely agree with you: it sucks to suck at maths because I believe this article brings very interesting information, information that could be very useful to me, that I simply cannot understand because I suck at maths.
Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…
You need to have basic intuition for what might be happening, the math is just a formality and frankly, is unnecessary beyond a very, very simple calculation.
You have to actually 'think' about behaviour a bit if you want to get it right, that's the hard part.
If you have something reasonable, then the conversions/control numbers can be worked out into a probability of success very quickly, and even then, just looking at them will give you a good idea if it worked or not.
The maths is a shiny lure for technical people, it gets us all excited as though there is some kind of truth behind it.
Earlier quoted context omitted.
My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.
It seems that any kind of data is mainly used to confirm the believes people already had and/or justify decisions already made.
This basic principle has far broader implications than website design and A/B testing. Managers at large corporations have learned to pull all sorts of levers to optimize the short term value of some metric (typically the one upon which their compensation depends) often in direct opposition to the long term interests of the corporation (and even the value of that metric beyond the next few quarters).
Is it even fine to use the distribution assumptions in the later analysis?
Looks like these assumptions combined with a higher conversion rate on day 2 for control is the main reason for the surprising result (control is obviously spread out).
Just because one design converts more than the other doesn't mean it's the design with optimal UX. I've seen many tests where the designs included already had faulty UX. This is why it's better to have a trained UX designer on your team who can fix basic flaws and present the best version of various designs for testing.
This can be quantified by plotting the incremental conversions observed by day x. We migh see a big initial lift that degrades over time. If it eventually degrades to zero, there are no truly incremental conversions, just pull-forward. But if we end up pulling forward a meaningful number of purchases by a month or more, that can be valuable to the business!
I wouldn't immediately jump to a complicated mathematical model to handle this situation, I would consider the business implications first and foremost.
I also urge anyone considering Bayesian methods for A/B testing to read up on the likelihood principle vs the strong repeated sampling principle (I documented my thoughts here [0]). Bayesian methods always satisfy the likelihood principle; frequentist methods always satisfy repeated sampling. In many situations both methods satisfy both principles, and then the two approaches will give similar answers. But based on many years doing A/B testing, I wouldn't give up repeated sampling lightly. Bayesian and frequentist methods are not blindly interchangeable.
On the other hand, if repeated sampling is not important in your use case, then by all means prefer the Bayesian approach! I just want people to consider the trade offs.
It sucks to suck at maths.
I'm not sure why you are being downvoted. I completely agree with you: it sucks to suck at maths because I believe this article brings very interesting information, information that could be very useful to me, that I simply cannot understand because I suck at maths.