Why multi-armed bandit algorithm is not "better" than A/B testing
visualwebsiteoptimizer.com
Why multi-armed bandit algorithm is not "better" than A/B testing
1–10 of 80 posts
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#2One glaring flaw in this article is that they're comparing a multi-arm bandit with a 10% exploration rate to a simple random split test with an implicit 50% exploration rate. Of course the split test will converge faster.
In their results when they compare a MAB with a 50% exploration rate to their split test you start to see a comparable amount of time to converge. Also they only show the results of one simulation with a lot of random in it. Given we're all stats nerds it would've been handy to see a box plot for each of the styles of simulation across multiple runs.
Having said all of that, my biggest concern around MAB as it's being sold is the lack of thought around the experiments. In the end it's just data and it doesn't mean anything without human intuition and preconceptions guiding it. Example: day of week, time of year/day, new page people are getting used to still, old one that garners lots of repeat traffic, etc. When running tests there's a lot more to consider besides just "whatever the data says".
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#3I would like to have seen more tests vs the MAB-90 algorithm as this appeared the most interesting one to me. Also the article appears to imply that A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument.
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#4Reading the conclusions it seems that each approach - a/b testing vs multi-armed bandit - has its merits. Maybe Visual Website Optimizer should be method agnostic and offer customers the choice of method?
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#5One glaring flaw in this article is that they're comparing a multi-arm bandit with a 10% exploration rate to a simple random split test with an implicit 50% exploration rate. Of course the split test will converge faster. In their results when they compare a MAB with a 50% exploration rate to their split test you start to see a comparable amount of time to converge. Also they only show the results of one simulation w…
We ran each simulation 25 times (with 10000 iterations each). And can share exact statistics in case you are interested. And yes, MAB with 50% exploration rate did perform similarly, but it didn't generalize when there were more than two versions.
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#6I would like to have seen more tests vs the MAB-90 algorithm as this appeared the most interesting one to me. Also the article appears to imply that A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument.
> A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument.
Why do you say so? Can you elaborate?
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#7Reading the conclusions it seems that each approach - a/b testing vs multi-armed bandit - has its merits. Maybe Visual Website Optimizer should be method agnostic and offer customers the choice of method?
Or use A/B testing to figure out which testing method customers respond to better....
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#8Without concrete details about the experiment setup and dataset on which it was run (and ideally the source code,so we can be sure there aren't any bugs) this is essentially meaningless.
Anyone can run "simulations" to prove anything. Providing just a summary table is of little use. I am not saying that the Wingify folks are trying to mislead people - just that this article doesn't have sufficient rigor to justify its conclusions.
OTOH, many CS papers, even published ones, don't provide source code or datasets so people can replicate the results, so perhaps this is the 'new normal' ;) .
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#9My take on bandit algorithms vs A/B testing:
If you have a question where the answer can be used to inform other decisions (e.g. Which is more important for our customers, price or support level?) then an A/B test will give you a clear answer that you can apply elsewhere.
If you don't really care what the answer is and just want the best option (e.g. should the hero shot be looking up or down) then a bandit is the way to go.
Re: Why multi-armed bandit algorithm is not "better" than A/B testing
#10Without concrete details about the experiment setup and dataset on which it was run (and ideally the source code,so we can be sure there aren't any bugs) this is essentially meaningless. Anyone can run "simulations" to prove anything. Providing just a summary table is of little use. I am not saying that the Wingify folks are trying to mislead people - just that this article doesn't have sufficient rigor to justify it…
Here's the code (quick-and-dirty): http://pastie.org/4007859
I had double-checked the code, but it is quite possible that I made an oversight somewhere.