Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
1–10 of 56 posts
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#2What, now it's indirect argument over HN?
Dear UX specialists with no knowledge of statistics: you can use the MAB algo with 2 choices, no problem. And it is a better way of getting 'the right choice'.
Dear statisticians: there's more to life (and to UX) than A/Bing (or MABing) everything
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#3Next: "why we're leaving bandit and A/B testing"
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#4Let's settle this with science rather than rhetoric. I'd like to do some proper comparisons between bandit algorithms and A/B testing. Unfortunately we haven't been saving time series data at Myna, so we don't have any test data. If anyone has time series data from an A/B test, and is happy to donate it to the cause, please get in touch (email in profile).
Updated for clarity.
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#5So am I getting this right?
(Edit) So the two steps to run are:
1. Run a traditional A/B test until 95% confidence is reached. This is full exploration.
2. Then, switch to the MAB after that, showing the better performing variant most of the time. As time increases, the display of the worse performing variants decreases.
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#6Let's settle this with science rather than rhetoric. I'd like to do some proper comparisons between bandit algorithms and A/B testing. Unfortunately we haven't been saving time series data at Myna, so we don't have any test data. If anyone has time series data from an A/B test, and is happy to donate it to the cause, please get in touch (email in profile). Updated for clarity.
First off, I'm all for settling this with data and math.
However, I'm afraid that some practical matters were lost in the previous discussions. As I see it, things happened somewhat like this: 1) A/B testing starts to become a Big Deal; 2) There's a flurry of articles about A/B testing of various quality; 3) Lots of people implement A/B testing, mostly in a poor way; 4) The 20 lines of code article, which would probably really help most people who haven't done A/B testing correctly; 5) Intense discussion about correctness and efficacy that won't impact those people.
I think getting to the bottom of this is important, but I think the multi-armed bandit article that started this would do far more good than harm for the droves of people trying and largely failing to get good data from their poorly done A/B testing.
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#7Let's settle this with science rather than rhetoric. I'd like to do some proper comparisons between bandit algorithms and A/B testing. Unfortunately we haven't been saving time series data at Myna, so we don't have any test data. If anyone has time series data from an A/B test, and is happy to donate it to the cause, please get in touch (email in profile). Updated for clarity.
Rather than try to mine historical data, run an experiment to pit UCB against Neyman-Pearson inference. For some A/B tests, split the users into two groups. Treatment A is A/B testing, treatment B is UCB.
In A/B testing, follow appropriate A/B testing procedures: Pick a sample size prior to the experiment that gives you appropriate power, or use Armitage's rule for optimal test termination. (Email me if you're interested, I'm happy to send over papers/scan relevant pages from his book). However , it's probably best to use a fixed sample size, as that is what most real life A/B test practitioners use. Picking the sample size can be a bit tricky, but as a rule of thumb, pick something that is large in enough to dectect differences in treatments as small as 1%age point.
In the treatment group B, use the UCB1 procedure. Subject the users to whichever design UCB1 picks, and continue with the learning.
Do not share any information between treatment groups A and B.
Run these tests for a sufficient amount of time over a largish number of clients, and then use permutation tests to determine which treatment, UCB1 vs Neyman-Pearson, performs better.
In all the simulations I've seen, UCB performs simple A/B testing, but it would be great to see some empirical evidence as well.
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#8So am I getting this right? (Edit) So the two steps to run are: 1. Run a traditional A/B test until 95% confidence is reached. This is full exploration. 2. Then, switch to the MAB after that, showing the better performing variant most of the time. As time increases, the display of the worse performing variants decreases.
This article suggests another option: Run UCB1 (a specific MAB variation) from day one and you'll get benefits of MAB without limitations of A/B testing or mathematical fallacies of the prior 20 lines of code MAB for n-cell tests.
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#9Let's settle this with science rather than rhetoric. I'd like to do some proper comparisons between bandit algorithms and A/B testing. Unfortunately we haven't been saving time series data at Myna, so we don't have any test data. If anyone has time series data from an A/B test, and is happy to donate it to the cause, please get in touch (email in profile). Updated for clarity.
Rather than try to mine historical data, run an experiment to pit UCB against Neyman-Pearson inference. For some A/B tests, split the users into two groups. Treatment A is A/B testing, treatment B is UCB. In A/B testing, follow appropriate A/B testing procedures: Pick a sample size prior to the experiment that gives you appropriate power, or use Armitage's rule for optimal test termination. (Email me if you're intere…
Most websites lack sufficient traffic to reach statistical significance in a short time frame. Sure, Google and Facebook can run a test and get real results in a day (or even hour(s)), but the rest of us need weeks or months to do things properly
Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
#10What is is the half-time rate of this debate?
300 days or less?
http://news.ycombinator.com/item?id=2831455
Sadly, the option to drop this problem over Germany doesn't exist any more.[1]
[1] http://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_mo...