Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

91–100 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#91

Post by hoddez »

There is at least one caveat with multi-armed bandit testing. It assumes that the site/app remains constant over the entire experiment. This is often not the case or feasible, especially for websites with large teams deploying constantly.

When your site is constantly changing in other ways, dynamically changing odds can cause a skew because you could give more of A than B during a dependent change, so you have to normalize for that somehow. A/B testing doesn't have this issue because the odds are constant over time.

Re: 20 lines of code that beat A/B testing (2012)

#92

Post by mplewis »

davkap92 wrote: post #40
ryporter wrote: post #36

This is a good overview of the multi-arm bandit problem [1], but the author is far too dismissive of A/B Testing. First of all, the suggested approach isn't always practical. Imagine that you are testing an overhaul of your website. Do you want daily individual visitors to keep flipping back and forth as the probabilities change? I'm not sure if the author is really suggesting his approach would be a better way to ru…

Regarding your first point, not sure if author covered it, but in Google Content Experiments with multi bandit approach cookies are stored, so user who sees variation b will keep seeing b while the experiment is running

This is the approach every good AB testing service/framework uses.

Re: 20 lines of code that beat A/B testing (2012)

#93

Post by mangeletti »

adevine wrote: post #86
mangeletti wrote: post #68

Earlier quoted context omitted.

The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand... I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained…

> I wanted Optimizely to say it was 100% significant That's not how statistical significance works...

You should contact Optimizely and let them know right now.

https://www.optimizely.com/contact/

They'll need to reeducate their statisticians right away!

Re: 20 lines of code that beat A/B testing (2012)

#94

Post by ced »

rco8786 wrote: post #63
mangeletti wrote: post #60

Earlier quoted context omitted.

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll f…

You really should question the statistical significance of those numbers.

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia:

    using Distributions

    b_old = Beta(66+1, 6392-66+1)
    b_yel = Beta(83+1, 6362-83+1)

    N = 1000000
    # Sample from both distributions, count the fraction of samples that are better
    sum(rand(b_old, N) .> rand(b_yel, N)) / N
This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers to a simple question.

Re: 20 lines of code that beat A/B testing (2012)

#96

Post by llull »

Bandits are great, but using the theory correctly can be difficult (and if accidentally incorrectly applied then ones results can easily become pathologically bad). For instance, the standard stochastic setup requires that learning instances are presented in an iid manner. This may not be true for website visitors, for example different behaviour at different times of day (browsing or executing) or timezone driven differing cultural responses. There is never a simple, magic solution for these things.

Re: 20 lines of code that beat A/B testing (2012)

#97

Post by searine »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

>I am free to just spew out content regardless of its quality.

Oh, that's a great goal to have...

Re: 20 lines of code that beat A/B testing (2012)

#98

Post by CPLX »

ced wrote: post #94
rco8786 wrote: post #63

Earlier quoted context omitted.

You really should question the statistical significance of those numbers.

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers…

Why is that answer so different? It's saying there's a 90%+ chance that the new one is better. That's also his answer.

Re: 20 lines of code that beat A/B testing (2012)

#99

Post by wlesieutre »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

> The first group would have the entirety of their grade based on the creativity of a single piece they submit. The second group was graded on only the total number of pounds of clay they threw.

I feel like that works partly because an important part of practice the feedback loop between continually practicing and having a sense of whether you did well or not.

Your strategy of not evaluating your own work sounds a bit like mushing clay into shapes with a blindfold on and then tossing it in kiln before you even check whether or not it's shaped like a pot. The users can sort through them later!

If the end goal is just ending up with a volume of work that's been culled down to the better ones, I guess you still get that. But it's inherently different from the Thinking Fast and Slow example where they're in a class and the goal is to learn and get better, rather than see who's made the nicest pot by the end of the semester.

Re: 20 lines of code that beat A/B testing (2012)

#100

Post by sixtypoundhound »

Novices also tend to gravitate towards "end-game" business metrics which have a lot more inherent variation than simple operational indicators.

For example - optimizing a content site for AdSense; many folks would gravitate to AdSense $$ as the target metric, which is admittedly an intuitive solution (since that's how you're ultimately getting paid).

But if you think about it....

AdSense Revenue =>

(1 - Bounce Rate) x Pages / Visit x % ads clicked x CPC

Bounce rate is binomial probability with a relatively high p-value (15%+), thus you can get statistically solid reads on results with a relatively small sample.

Pages / Visit is basically the aggregate of a Markov chain (1 - exit probability); also relatively stable.

% ads clicked - binomial probability with low p-value; large samples becomes important

$ CPC - so the ugly thing here is there's a huge range in the value of a click... often as low as $.05 for a casual mobile phone click or $30 for a well qualified financial or legal click (think retargeting, with multiple bidders). And you're usually dealing with a small sample of clicks (since the average % CTR is very low). So HUGE natural variation in results. Oh, and Google likes to penalty price sites with a large rapid increase in click-through-rate (for a few days), so your short term CPC may not resemble what you would earn in steady-state.

So while it may make ECONOMIC sense to use test $ RPM as a metric, you've injected tremendous variation into the test. You can accurately read bounce rate, page activity, and % click-through on a much smaller sample and feel comfortable making a move if you're confident nothing major has changed in terms of the ad quality (and CPC value) you will get.

Post reply on HN