20 lines of code that beat A/B testing every time
stevehanov.ca
20 lines of code that beat A/B testing every time
1–10 of 152 posts
Re: 20 lines of code that beat A/B testing every time
#2This is the most important critique of A/B testing. It far outweighs the traditional hoopla about simultaneous inference and Bonferonni corrections.
Epsilon greedy does well on k-armed bandit problems, but in most applications you likely can do significantly better by customizing the strategy to individual users. That's a contextual bandit and there are simple strategies that to pretty well here too. For instance:
http://hunch.net/~exploration_learning/main.pdf
http://web.mit.edu/hauser/www/Papers/Hauser_Urban_Liberali_B...
Re: 20 lines of code that beat A/B testing every time
#3This sounds really interesting, I might have been close to building this before without realizing. :)
Seems really easy to setup as well, will be interesting to hear any counter-argument in the comments.
Re: 20 lines of code that beat A/B testing every time
#4Seems like what Myna is doing: http://mynaweb.com/
Re: 20 lines of code that beat A/B testing every time
#5There are appropriate solutions to the multi-armed bandit problem, and a wealth of literature out there, however this is not one of those solutions.
Here's a simple thought experiment to show that this will not 'beat A/B testing every time.' Imagine you have two designs, one has a 100% conversion rate, one has a 0% conversion rate. Simple A/B testing will allow you to pick the the winning example. Whereas this solution is still picking the 0% design 10% of the time.
For some other implementations check out the following links:
For Dynamic Resampling:
http://jmlr.csail.mit.edu/papers/volume3/auer02a/auer02a.pdf
For Optimal Termination Time:
http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...
Re: 20 lines of code that beat A/B testing every time
#6The 'set and forget' aspect of this is appealing. I've sometimes wondered if you could automate the whole thing, including option generation. If you can define good enough mutation functions you could have your features literally evolve over time, without developer input. You'd need a lot of throughput to get reasonable evolution rates though. Jacking up the mutation rate won't help because really big mutations will break the layout.
It's almost certainly impracticable, but fun to think about.
Re: 20 lines of code that beat A/B testing every time
#7This sounds really interesting, I might have been close to building this before without realizing. :) Seems really easy to setup as well, will be interesting to hear any counter-argument in the comments.
Seconded. As a non-mathsy marketing/biz guy, my response was "whoa, sounds great.", but with the caveat that my response to the actual statistical math side was "Derp."
I'm really looking forward to hearing comments from people With Actual Maths!
Re: 20 lines of code that beat A/B testing every time
#8This data isn't fully valuable unless you know what alternate behaviour the users are performing (instead of clicking the alternate-coloured buttons). They still could be staying on your site, just not following the same funnel. They could be return visitors as well.
Re: 20 lines of code that beat A/B testing every time
#9Does anyone know of any split test products on the market that do this?
Re: 20 lines of code that beat A/B testing every time
#10There are appropriate solutions to the multi-armed bandit problem, and a wealth of literature out there, however this is not one of those solutions. Here's a simple thought experiment to show that this will not 'beat A/B testing every time.' Imagine you have two designs, one has a 100% conversion rate, one has a 0% conversion rate. Simple A/B testing will allow you to pick the the winning example. Whereas this soluti…
OK, in AB testing that same 0% design, you're showing it 50% of the time.
You seem to be saying "I'll AB test it just for a little, then weed out the 0% one. but in the case of this new algorithm, I'll let it run for a long time." That's not exactly fair. Not to mention, both algorithms would allow you to clearly see the 0% option sucks.