Run fewer, better A/B tests
edoconti.medium.com
Run fewer, better A/B tests
1–10 of 23 posts
Re: Run fewer, better A/B tests
#2I've seen a lot of really sophisticated data pipelines and testing frameworks at a lot of shops. I've seen precious few who were able to make well-considered product decisions based on the data.
Re: Run fewer, better A/B tests
#3I'm going to stick with multiarm bandit testing.
Re: Run fewer, better A/B tests
#4Between the emojis in the headings and the 2009 era memes, this was a bit of a cringy read. Also, the author seems to avoid at all costs going in depth about the actual implementation of OPE and I still don't quite understand how I would go about implementing it. Machine learning based on past A/B tests that finds similarities between the UI changes???
Re: Run fewer, better A/B tests
#5The challenge I've seen is to have a combination of good, small-scale Human-Centered Design research (watching people work,for instance) and good, large-scale testing. It can be really hard to learn the "why" from a/b tests otherwise.
Re: Run fewer, better A/B tests
#6Between the emojis in the headings and the 2009 era memes, this was a bit of a cringy read. Also, the author seems to avoid at all costs going in depth about the actual implementation of OPE and I still don't quite understand how I would go about implementing it. Machine learning based on past A/B tests that finds similarities between the UI changes???
Yea me too.
My biggest question is where do you get user data to run the simulation? Take the simple push example - if to date you’ve only sent pushes on day 1, and you want to explore day 2,3,4,5 etc…where does that user response data come from? It seems like you need to get the data, then you can simulate various permutations of a policy. But then why not just run multi arm bandit?
Re: Run fewer, better A/B tests
#7I'm going to stick with multiarm bandit testing.
This is a much better approach...
Re: Run fewer, better A/B tests
#8When I was the A/B test guy for Travelocity I was fortunate to have an excellent team. The largest bias we discovered is that our tests were executed with amazing precision and durability. My dedicated QA was the whining star that made that happen. Unfortunately when the resulting feature entered the site in production as a released feature there was always some defect, or some conflict, or some oversight. The actual business results would then under perform compared to the team’s analyzed prediction.
Re: Run fewer, better A/B tests
#9The notifications examples make me wonder what fundamental mistakes they are making.
People respond to change. If you A/B test, say, a new email headline, the change usually wins. Even if it isn't better. Just because it is different. Then you roll it out in production, look at it a few months later, and it is probably worse.
If you don't understand downsides like this, then A/B testing is going to have a lot of pitfalls that you won't even know that you fell into.
Re: Run fewer, better A/B tests
#10Recently I hear that booking.com is given as an example of a company that runs a lot of a/b tests. Anyone from booking reading this? How does it look from the inside, is it worth it to run hundreds at a time?