Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing every time

stevehanov.ca

Post reply on HN
Print view View original

31–40 of 152 posts

Re: 20 lines of code that beat A/B testing every time

#31

Post by krupan »

robertskmiles wrote: post #6

The 'set and forget' aspect of this is appealing. I've sometimes wondered if you could automate the whole thing, including option generation. If you can define good enough mutation functions you could have your features literally evolve over time, without developer input. You'd need a lot of throughput to get reasonable evolution rates though. Jacking up the mutation rate won't help because really big mutations will…

I'd love to see a website designed entirely by statistical machine learning :-)

Re: 20 lines of code that beat A/B testing every time

#32

Post by Swizec »

wisty wrote: post #23

A stevehanov.ca link? Wow, HN is getting classy again. Please more articles with code, equations and / well visualizations, and less upvoting of badly thought out infograpics (i.e. pretty numbers which would lose nothing by just being presented in a table) and far less self-help pseudo business articles please. +1 on an article does not mean "I agree". It means "I learnt something".

I upvoted you because I agree.

But I usually upvote stories based on how interesting they are and whether I feel like they were worth attention. Comments are a mix bag of agreeing, interesting and just well thought out arguments that make me think.

This is completely meta and kind of offtopic, so it should probably be downvoted, but I felt like saying it anyway.

Re: 20 lines of code that beat A/B testing every time

#33

Post by toemetoch »

Next step: close the loop.

Option 1: Let the next set of values (colors of the button in the example) themselves be generated after n runs.

Option 2: let users modify the css as part of a "customization " feature. I remember they did this before the new BBC site was officially launched a few years ago.

Re: 20 lines of code that beat A/B testing every time

#34

Post by heyitsnick »

Maybe i'm missing it (it's late), but nowhere in the article does it explain why 10% of the time it picks a choice at random ("explores"). In fact, the article basically argues why it's not needed (it self-rights if the wrong choice becomes temporarily dominant). It also doesn't explain why specifically it should be a 10% randomization.

Re: 20 lines of code that beat A/B testing every time

#35

Post by mattj »

medell wrote: post #27
rauljara wrote: post #12

This is an interesting technique, but it too has flaws. If there is a period of buzz and excitement surrounding your app, whatever design was most popular at that time will be rewarded accordingly, and accrue a high click through rate with tens of thousands of case. If you introduce a new superior design after the period of buzz has gone away, the new design may take a very long time to catch up. Even though it is cu…

Nice post. One hypothetical case it could end up serving the worse design more often is if you had times of the day where your users behave very differently due to time zones, i.e. Europe vs North America. Say you have a sports site and test a new soccer oriented layout vs an old baseball heavy one. In the day, the old baseball version wins easily. When NA goes to sleep it would serve up baseball to the Europeans unt…

This is actually pretty straightforward to overcome (and one of the real strengths of the bayesian approach). Rather than using the direct counts of success for each group, add a prior belief that americans will favor baseball and non-americans will favor soccer (you can experiment to determine this number).

You can now evaluate the results conditioned on each group (american / non-american).

Re: 20 lines of code that beat A/B testing every time

#37

Post by azernik »

heyitsnick wrote: post #34

Maybe i'm missing it (it's late), but nowhere in the article does it explain why 10% of the time it picks a choice at random ("explores"). In fact, the article basically argues why it's not needed (it self-rights if the wrong choice becomes temporarily dominant). It also doesn't explain why specifically it should be a 10% randomization.

The problem is that, depending on your initial settings and early results, certain settings could get such low payoff estimates that they're never tried at all (let's say one gets to 50% after one trial, and the rest all stay above 75%). You want to make sure that your solver adequately explores all choices.

Re: 20 lines of code that beat A/B testing every time

#38

Post by treeface »

IgorPartola wrote: post #18
rauljara wrote: post #12

This is an interesting technique, but it too has flaws. If there is a period of buzz and excitement surrounding your app, whatever design was most popular at that time will be rewarded accordingly, and accrue a high click through rate with tens of thousands of case. If you introduce a new superior design after the period of buzz has gone away, the new design may take a very long time to catch up. Even though it is cu…

Better than a forgetting factor, add a Kalman filter ( http://en.wikipedia.org/wiki/Kalman_filter ). This way you can trust your "new" data more than really "old" data, etc. The beauty of it is that it only adds three attributes to each data sample.

Could you expound on this a bit? What attributes would you have to add? How would you calculate scores?

Re: 20 lines of code that beat A/B testing every time

#39

Post by daeken »

heyitsnick wrote: post #34

Maybe i'm missing it (it's late), but nowhere in the article does it explain why 10% of the time it picks a choice at random ("explores"). In fact, the article basically argues why it's not needed (it self-rights if the wrong choice becomes temporarily dominant). It also doesn't explain why specifically it should be a 10% randomization.

A random choice is needed to allow people to give rewards to options other than the dominant. I'm sure this doesn't have to be random -- and I'd be curious to see the logic behind the 10% choice -- but you have to have something that gives the other options a chance.

Makes me wonder if the 10% number couldn't be changed to something that's a function of the number of rewards total; the longer it runs, the less variation there is and the more confident you are in the choice made.

Re: 20 lines of code that beat A/B testing every time

#40

Post by daeken »

TomGullen wrote: post #36
wdewind wrote: post #22

This does require a significantly larger sample size for accuracy though.

A larger sample size than A/B? Why?

I don't know for sure, but it (intuitively) seems to me that you'd require a larger sample size because only 10% of the time will it explore other options. With A/B testing, you've got equal odds to hit either option, so each step is independent of the last; with this, each step depends on the previous results and thus if you have one option jump out into the lead (for whatever reason) it'd make it less likely that the other gets a good sample.
Post reply on HN