Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

chrisstucchio.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#31

Post by btilly »

patio11 wrote: post #20

For what it's worth, I've been following this cross-Internet debate with more than a little professional interest. Cards on the table: I have coded A/B testing software, I frequently code and/or administer it for clients (often in ways which are provably suboptimal), and I am a dirty loyalty-free scientist-cum-capitalist-pig who would stab A/B testing in the back in a second if I thought there were an easier way to e…

Thank you for saying that. For people who want to dive in deeper, the discussion below http://news.ycombinator.com/item?id=4053739 is highly worthwhile as well.

And you are absolutely right. Even with bad assumptions and techniques, actually testing beats not testing by such a ridiculous margin that you need to.

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#32

Post by tel »

patio11 wrote: post #22
yummyfajitas wrote: post #21

Earlier quoted context omitted.

Pick A/B testing, bandit testing, whatever -- any option in the set, even with poor algorithms and/or the easiest errors I can think of, strictly dominate not testing at all. So I submitted this post and went off to boxing. On the train ride back, I thought "I hope I don't make people think they shouldn't A/B test. And at the same time you were writing this post, I added a conclusion to my blog post saying the same t…

Unless I misunderstand bandit algorithms, there's a trivial modification that makes the actual, practical administration of them essentially identical to A/B testing with regards to when you can rip out code. If A smashes B, then bandit will converge in a very obvious manner on A, and you pick it and delete the B code branch, accepting future regret from the possibility that B was in fact better as a cost of doing bu…

I think that assuming stationary behavior (bad assumption) then given enough time either A will smash B or B will smash A. As in, it's almost impossible that they are exactly equal in performance and given enough impressions even the smallest change will be optimized for.

I think that's a bad thing, though. A single conversion metric optimization is only good for some (large or medium) effect size. After a point, criterions like "I kind of like A" are much more important.

In A/B testing you see this when your ambitious testing campaign returns "insignificant". In MAB you see it when two choices run at roughly 50/50 enrollment for a long period of time.

So on these two extremes, I think practical use of A/B and MAB should be roughly identical. In the middle ground where A is usefully but not incredibly better than B, I feel they must differ.

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#33

Post by noelwelsh »

patio11 wrote: post #14
noelwelsh wrote: post #4

Let's settle this with science rather than rhetoric. I'd like to do some proper comparisons between bandit algorithms and A/B testing. Unfortunately we haven't been saving time series data at Myna, so we don't have any test data. If anyone has time series data from an A/B test, and is happy to donate it to the cause, please get in touch (email in profile). Updated for clarity.

Write out what .CSV columns you need and what formats they need to be in, and I will happily get you this for a handful of A/B tests. (Though probably not faster than late July. As much as I love A/B testing there is the small matter of a wedding and honeymoon to throw a wee bit of a wrench into my near term schedule...)

Thanks. I'll be in touch.

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#34

Post by vii »

patio11 wrote: post #20

For what it's worth, I've been following this cross-Internet debate with more than a little professional interest. Cards on the table: I have coded A/B testing software, I frequently code and/or administer it for clients (often in ways which are provably suboptimal), and I am a dirty loyalty-free scientist-cum-capitalist-pig who would stab A/B testing in the back in a second if I thought there were an easier way to e…

In practice, if we were going to talk about that -- variations which perform statistically significantly better one day may perform worse the next. For example, because happy people like to click on one variation versus another which is preferred by stressed people.

Keep A-B tests running even after you've decided the best variation to make sure that the uplift you've observed is real! See http://john.freml.in/ab-testing-significance

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#35

Post by dllthomas »

patio11 wrote: post #22
yummyfajitas wrote: post #21

Earlier quoted context omitted.

Pick A/B testing, bandit testing, whatever -- any option in the set, even with poor algorithms and/or the easiest errors I can think of, strictly dominate not testing at all. So I submitted this post and went off to boxing. On the train ride back, I thought "I hope I don't make people think they shouldn't A/B test. And at the same time you were writing this post, I added a conclusion to my blog post saying the same t…

Unless I misunderstand bandit algorithms, there's a trivial modification that makes the actual, practical administration of them essentially identical to A/B testing with regards to when you can rip out code. If A smashes B, then bandit will converge in a very obvious manner on A, and you pick it and delete the B code branch, accepting future regret from the possibility that B was in fact better as a cost of doing bu…

This, I think.

Once you get your expected regret below the cost of maintaining the old code, rip out the old code!

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#36

Post by btilly »

noelwelsh wrote: post #4

Let's settle this with science rather than rhetoric. I'd like to do some proper comparisons between bandit algorithms and A/B testing. Unfortunately we haven't been saving time series data at Myna, so we don't have any test data. If anyone has time series data from an A/B test, and is happy to donate it to the cause, please get in touch (email in profile). Updated for clarity.

Unfortunately most of the A/B tests that I am involved with at the moment simply do not look anything like what you'd want for a traditional bandit approach. For a start the data is fundamentally 2-dimensional. I am testing by user, and users may take days to hit a conversion metric. Furthermore a large chunk of the population has to go in the test before I get any data at all about how any of them will perform.

This is not an uncommon scenario when testing an email program. And points to operational reasons why A/B testing may be preferred even if you believe that it is statistically worse.

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#37

Post by btilly »

raverbashing wrote: post #2

What, now it's indirect argument over HN? Dear UX specialists with no knowledge of statistics: you can use the MAB algo with 2 choices, no problem. And it is a better way of getting 'the right choice'. Dear statisticians: there's more to life (and to UX) than A/Bing (or MABing) everything

Dear random internet poster. This argument has happened several times already on this site. I recommend reading previous discussion at http://news.ycombinator.com/item?id=4052997 and http://news.ycombinator.com/item?id=4040022 before being sure that this blog is obviously the final say.

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#38

Post by lucisferre »

yummyfajitas wrote: post #21
patio11 wrote: post #20

For what it's worth, I've been following this cross-Internet debate with more than a little professional interest. Cards on the table: I have coded A/B testing software, I frequently code and/or administer it for clients (often in ways which are provably suboptimal), and I am a dirty loyalty-free scientist-cum-capitalist-pig who would stab A/B testing in the back in a second if I thought there were an easier way to e…

Pick A/B testing, bandit testing, whatever -- any option in the set, even with poor algorithms and/or the easiest errors I can think of, strictly dominate not testing at all. So I submitted this post and went off to boxing. On the train ride back, I thought "I hope I don't make people think they shouldn't A/B test. And at the same time you were writing this post, I added a conclusion to my blog post saying the same t…

Wouldn't bad A/B testing be a 50/50 solution. Is making decisions on data without statistical significance any better than throwing darts blind?

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#39

Post by janzer »

lucisferre wrote: post #38
yummyfajitas wrote: post #21

Earlier quoted context omitted.

Pick A/B testing, bandit testing, whatever -- any option in the set, even with poor algorithms and/or the easiest errors I can think of, strictly dominate not testing at all. So I submitted this post and went off to boxing. On the train ride back, I thought "I hope I don't make people think they shouldn't A/B test. And at the same time you were writing this post, I added a conclusion to my blog post saying the same t…

Wouldn't bad A/B testing be a 50/50 solution. Is making decisions on data without statistical significance any better than throwing darts blind?

There is a large gap between true statistical significance and a fair coin toss. Bad testing (of whatever flavor A/B, MAB, etc.) is likely to land somewhere in that gap. Most likely worse off than proper testing but also quite likely better than tossing the coin or throwing darts.

Re: Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)

#40

Post by ricardobeat »

patio11 wrote: post #20

For what it's worth, I've been following this cross-Internet debate with more than a little professional interest. Cards on the table: I have coded A/B testing software, I frequently code and/or administer it for clients (often in ways which are provably suboptimal), and I am a dirty loyalty-free scientist-cum-capitalist-pig who would stab A/B testing in the back in a second if I thought there were an easier way to e…

I think concern #2 was addressed in the original article: add a "fade out" threshold so that results are fixed to some time span.
Post reply on HN