Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing every time

stevehanov.ca

Post reply on HN
Print view View original

51–60 of 152 posts

Re: 20 lines of code that beat A/B testing every time

#51

Post by nickzoic »

mattj wrote: post #35
medell wrote: post #27

Earlier quoted context omitted.

Nice post. One hypothetical case it could end up serving the worse design more often is if you had times of the day where your users behave very differently due to time zones, i.e. Europe vs North America. Say you have a sports site and test a new soccer oriented layout vs an old baseball heavy one. In the day, the old baseball version wins easily. When NA goes to sleep it would serve up baseball to the Europeans unt…

This is actually pretty straightforward to overcome (and one of the real strengths of the bayesian approach). Rather than using the direct counts of success for each group, add a prior belief that americans will favor baseball and non-americans will favor soccer (you can experiment to determine this number). You can now evaluate the results conditioned on each group (american / non-american).

Or if you don't want to add in a prior specific belief about americans vs. non-americans, just keep one counter per option per continent-of-source-IP (or whatever) and the learning algorithm should work it out on its own. Of course, if you use too many bins then learning is going to take far too long.

Re: 20 lines of code that beat A/B testing every time

#52

Post by olefoo »

krupan wrote: post #31
robertskmiles wrote: post #6

The 'set and forget' aspect of this is appealing. I've sometimes wondered if you could automate the whole thing, including option generation. If you can define good enough mutation functions you could have your features literally evolve over time, without developer input. You'd need a lot of throughput to get reasonable evolution rates though. Jacking up the mutation rate won't help because really big mutations will…

I'd love to see a website designed entirely by statistical machine learning :-)

Google leans pretty heavily on machine learning for even trivial design choices http://stopdesign.com/archive/2009/03/20/goodbye-google.html mentions the infamous testing of 41 shades of bue to decide which was the correct one.

But truly someone has to design the learning system, at some level you will always have design, if it is the design of how the design is to adapt to it's environment.

Re: 20 lines of code that beat A/B testing every time

#53

Post by btilly »

This is thought-provoking, which is good. However there are significant issues with the approach.

1. Real world performance varies over time. For instance there are typically daily, weekly and monthly conversion rate fluctuations. Not an issue for A/B testing, but a big issue for this approach if a random switch in direction happens at the same time that conversion fluctuations happen to head in a good direction.

2. (This is really a special case of #1 - but a very, very important special case.) This approach creates long-lasting interaction effects between tests and independent changes. That requires explanation. Suppose you're running a test. Version A is somewhat better. But version B is temporarily looking slightly better when you make a significant improvement to your website (maybe you started another test that works). Now you're adding a lot of good traffic to version B (good because of the other change) and very little of the new good traffic to version A. This new version B traffic soundly beats your old version A traffic. This correlation between time and website performance will continue until the old version A traffic is completely swamped by new version A traffic. With only 5% of your traffic going to version A, this can easily take 100x as long as your test has been running - or more. (Properly constructed A/B tests do not suffer this statistical anomaly.)

3. Code over time gets messy. One of the most important characteristics of A/B testing is that you can delete the mess and move on. With this approach you can't - it just hangs around adding to your technical debt.

4. Businesses are complex, and often have multiple measures they would like to balance. For instance in a recent test, conversion to click was hurt, conversion to a person who clicked 5x was helped. A/B testing let us notice that something weird was going on and think about what we really cared about. This automated approach would make a decision and could have hidden a real problem.

5. Many tests perform differently on existing users and new users. A/B testing with proper cohort analysis can let you tease this out and decide accordingly. This approach doesn't give you that kind of sophistication.

Re: 20 lines of code that beat A/B testing every time

#54

Post by emiliobumachar »

rauljara wrote: post #12

This is an interesting technique, but it too has flaws. If there is a period of buzz and excitement surrounding your app, whatever design was most popular at that time will be rewarded accordingly, and accrue a high click through rate with tens of thousands of case. If you introduce a new superior design after the period of buzz has gone away, the new design may take a very long time to catch up. Even though it is cu…

I don't think this flaw is real.

If "whatever design was most popular at that time" has a billion trials and 90% successes, and "a new superior design" has 100 trials and 97% successes (97 out of 100), than the new design is favored by the algorithm. No need to "catch up" to the absolute number of successes.

Re: 20 lines of code that beat A/B testing every time

#55

Post by btilly »

timr wrote: post #45
wisty wrote: post #23

A stevehanov.ca link? Wow, HN is getting classy again. Please more articles with code, equations and / well visualizations, and less upvoting of badly thought out infograpics (i.e. pretty numbers which would lose nothing by just being presented in a table) and far less self-help pseudo business articles please. +1 on an article does not mean "I agree". It means "I learnt something".

Bandit optimization has been discussed previously on HN: http://news.ycombinator.com/item?id=2831455 The problem with this article is that it's a) not very detailed, and b) the conclusion is linkbait. Bandit optimization is a useful tool, but it has drawbacks, and it's not always better than A/B testing. In particular, bandit approaches take longer to converge (on average), and don't give you reliable ways to know wh…

The really scary drawback is what happens if the bandit prefers a suboptimal choice at the same time that you make an independent improvement in your website. Then the bandit is going to add a lot of data for that variation, all of which looks really good for reasons that have nothing to do with what it is supposed to be testing.

This type of error (which can happen very easily on a website going through continuous improvement) can take a very long time to recover from.

A/B tests do not have an issue with this because all versions will have similar mixes of data from before and after the improvement.

Re: 20 lines of code that beat A/B testing every time

#57

Post by timr »

wpietri wrote: post #50
timr wrote: post #45

Earlier quoted context omitted.

Bandit optimization has been discussed previously on HN: http://news.ycombinator.com/item?id=2831455 The problem with this article is that it's a) not very detailed, and b) the conclusion is linkbait. Bandit optimization is a useful tool, but it has drawbacks, and it's not always better than A/B testing. In particular, bandit approaches take longer to converge (on average), and don't give you reliable ways to know wh…

Agreed. And I'd add that for me, the point of A/B testing is to learn something. We're not just interested in whether A or B is better; we're interesting in getting better at doing what we do. Studying the A/B results is half the fun.

"We're not just interested in whether A or B is better; we're interesting in getting better at doing what we do."

Absolutely. That's my biggest objection to bandit methods, but it's also the fuzziest objection, and the one least likely to appeal to hyper-analytical people. There's a strong temptation (as we can see from this article) is to treat bandit optimization as a black box that just spits out an infallible answer (i.e. as a Lazy Button).

It's the same human tendency that has led to "you should follow me on twitter" to be one of the more common n-grams on the interwebs (even though it probably never worked for more than Dustin Curtis, and likely causes a counter-intuitive backlash now).

Re: 20 lines of code that beat A/B testing every time

#58

Post by nerdo »

daeken wrote: post #39
heyitsnick wrote: post #34

Maybe i'm missing it (it's late), but nowhere in the article does it explain why 10% of the time it picks a choice at random ("explores"). In fact, the article basically argues why it's not needed (it self-rights if the wrong choice becomes temporarily dominant). It also doesn't explain why specifically it should be a 10% randomization.

A random choice is needed to allow people to give rewards to options other than the dominant. I'm sure this doesn't have to be random -- and I'd be curious to see the logic behind the 10% choice -- but you have to have something that gives the other options a chance. Makes me wonder if the 10% number couldn't be changed to something that's a function of the number of rewards total; the longer it runs, the less variat…

That would be Epsilon-decreasing strategy or VDBE:

http://en.wikipedia.org/wiki/Multi-armed_bandit#Semi-uniform...

Re: 20 lines of code that beat A/B testing every time

#59

Post by r00fus »

krupan wrote: post #31
robertskmiles wrote: post #6

The 'set and forget' aspect of this is appealing. I've sometimes wondered if you could automate the whole thing, including option generation. If you can define good enough mutation functions you could have your features literally evolve over time, without developer input. You'd need a lot of throughput to get reasonable evolution rates though. Jacking up the mutation rate won't help because really big mutations will…

I'd love to see a website designed entirely by statistical machine learning :-)

What's to say those spam websites aren't exactly that?

Re: 20 lines of code that beat A/B testing every time

#60

Post by bbatchelder »

aaronjg wrote: post #5

There are appropriate solutions to the multi-armed bandit problem, and a wealth of literature out there, however this is not one of those solutions. Here's a simple thought experiment to show that this will not 'beat A/B testing every time.' Imagine you have two designs, one has a 100% conversion rate, one has a 0% conversion rate. Simple A/B testing will allow you to pick the the winning example. Whereas this soluti…

Possibly pedantic - but it would only show the loser 5% of the time. 10% of the time it picks a random design, including the winner.
Post reply on HN