Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

11–20 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#11

Post by timr »

isoprophlex wrote: post #5

As one of the comments below the article states, the probabilistic alternative to epsilon-greedy is worth exploring ad well. Take the "bayesian bandit", which is not much more complex but a lot more powerful. If you crave more bandits: https://jamesrledoux.com/algorithms/bandit-algorithms-epsilo...

Just a warning to those people who are potentially implementing it: it doesn't really matter. The blog author addresses this, obliquely (says that the simplest thing is best most of the time), but doesn't make it explicit.

In my experience, obsessing on the best decision strategy is the biggest honeypot for engineers implementing MAB. Epsilon-greedy is very easy to implement and you probably don't need anything more. Thompson sampling is a pain in the butt, for not much gain.

Re: Lines of code that beat A/B testing (2012)

#13

Post by douglee650 »

tracerbulletx wrote: post #8

A lot of sites don't have enough traffic to get statistical significance with this in a reasonable amount of time and it's almost always testing a feature more complicated than button color where you aren't going to have more than the control and variant.

Yes wondering what the confidence intervals are.

Re: Lines of code that beat A/B testing (2012)

#14

Post by lelandfe »

zahlman wrote: post #6

Nothing shows on this page without JavaScript except for the header and a grey background. A bit strange for a blog.

Pre-CSS grid masonry layout. Author hides the content with CSS, and JS reveals it, to avoid a flash.

CSS to make it noscript friendly: `.main { visibility: visible !important; max-width: 710px; }`

Re: Lines of code that beat A/B testing (2012)

#16

Post by crazygringo »

No, multi-armed bandit doesn't "beat" A/B testing, nor does it beat it "every time".

Statistical significance is statistical significance, end of story. If you want to show that option B is better than A, then you need to test B enough times.

It doesn't matter if you test it half the time (in the simplest A/B) or 10% of the time (as suggested in the article). If you do it 10% of the time, it's just going to take you five times longer.

And A/B testing can handle multiple options just fine, contrary to the post. The name "A/B" suggests two, but you're free to use more, and this is extremely common. It's still called "A/B testing".

Generally speaking, you want to find the best option and then remove the other ones because they're suboptimal and code cruft. The author suggests always keeping 10% exploring other options. But if you already know they're worse, that's just making your product worse for those 10% of users.

Re: Lines of code that beat A/B testing (2012)

#17

Post by jbentley1 »

Multi-armed bandits make a big assumption that effectiveness is static over time. What can happen is that if they tip traffic slightly towards option B at a time when effectiveness is higher (maybe a sale just started) B will start to overwhelmingly look like a winner and get locked in that state.

You can solve this with propensity scores, but it is more complicated to implement and you need to log every interaction.

Re: Lines of code that beat A/B testing (2012)

#18

Post by asdasdsddd »

tantalor wrote: post #10
asdasdsddd wrote: post #2

Multi arm bandits are fine but their limited to tests where its ok to switch users between arms frequently and tests that have more power

> where its ok to switch users between arms frequently It's not hard to keep track of which arm any given user was exposed to in the first run, and then repeat it.

There are often product limitations

Re: Lines of code that beat A/B testing (2012)

#19

Post by __MatrixMan__ »

After this:

> hundreds of the brightest minds of modern civilization have been hard at work not curing cancer. Instead, they have been refining techniques for getting you and me to click on banner ads

I was really hoping this would slowly develop into a statistical technique couched in terms of ad optimization but actually settling in on something you might call ATCG testing (e.g. the biostatistics methods that one would indeed use to cure cancer).

Post reply on HN