Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

101–110 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#101

Post by kens »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I tried to find the original source of the quality-vs-quantity pottery class story a while back. I think it originates in the book "Art and Fear" but in that book it reads like a parable rather than a factual event. I'm highly suspicious of whether this event actually happened. Anyone have solid evidence?

Re: 20 lines of code that beat A/B testing (2012)

#102

Post by KingMob »

butler14 wrote: post #28
spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.

To ignore seasonality requires assuming it has roughly the same effect on both groups. If they move in different directions or by very different amounts, you can't actually ignore it.

Simple example: you're comparing your standard site's theme with a new pumpkin orange theme. In October, the pumpkin orange theme might go great, whereas in December, it might perform worse. There's clearly a seasonality interaction you need to account for.

What you're thinking of is more like testing e-commerce button colors right after Christmas. After Christmas, it's likely that all your groups perform worse, but equally so.

Re: 20 lines of code that beat A/B testing (2012)

#103

Post by tamana »

mangeletti wrote: post #93
adevine wrote: post #86

Earlier quoted context omitted.

> I wanted Optimizely to say it was 100% significant That's not how statistical significance works...

You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!

Optimizely did an A/B test that showed that customers respond better to rounded-up numbers. ;-) the real world is messy.

Years ago my team's statistician did a competitive review of various AB test apps, and reported various ways in which the UIs make statistically invalid statements to the user.

Re: 20 lines of code that beat A/B testing (2012)

#104

Post by xixi77 »

mangeletti wrote: post #93
adevine wrote: post #86

Earlier quoted context omitted.

> I wanted Optimizely to say it was 100% significant That's not how statistical significance works...

You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!

They probably do, but why would they want to do so, when doing it would make their results appear less conclusive?

Re: 20 lines of code that beat A/B testing (2012)

#105

Post by chias »

I like the premise of this a lot, but it seems to me that the setting that the author chose (some UI element of a website) is one of the worst possible settings for this: what matters a whole lot more than if your button is red or green or blue is some modicum of consistency.

If you're constantly changing the button color, size, location, whatever... that is an awful experience in and of itself, is it not? If the Amazon "buy now" button changed size / shape / position / color every time I went to buy something, I would get frustrated with it pretty quickly.

Re: 20 lines of code that beat A/B testing (2012)

#106

Post by norkakn »

mangeletti wrote: post #93
adevine wrote: post #86

Earlier quoted context omitted.

> I wanted Optimizely to say it was 100% significant That's not how statistical significance works...

You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!

The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about the size of the differences. It also can't be 100. The normal distribution has an infinite range, so a finite limit can never capture 100% of it.

Optimizely is in a rough spot. People don't like having to think through experimental design, and they are really, really bad at reasoning about p-values. To try to fix the people part, they came out with the sequential stopping rule stuff (their "stats engine), but they never really published much justifying it. The other alternative would be to move the experiments into a Bayesian framework, but that has a lot of it's own problems. When they acquired Synference, that was one of the likely directions to take (along with offering bandits), but that didn't work out and those guys have since left.

Re: 20 lines of code that beat A/B testing (2012)

#107

Post by Quinner »

mangeletti wrote: post #93
adevine wrote: post #86

Earlier quoted context omitted.

> I wanted Optimizely to say it was 100% significant That's not how statistical significance works...

You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!

There's a reason you'll only see their number say >99%

Re: 20 lines of code that beat A/B testing (2012)

#108

Post by tswartz »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

Good point! What do you use to do in-app A/B testing?

Re: 20 lines of code that beat A/B testing (2012)

#109

Post by cle »

This is only appropriate in certain situations. There are many business situations in which it's more appropriate to run a traditional A/B test and carefully examine and understand the results before making a business decision. Always blindly accepting the results of a bandit is going to explode in your face at some point.

There is no silver bullet, no free lunch. There is no algorithm that will beat understanding your domain and carefully analyzing your data.

Re: 20 lines of code that beat A/B testing (2012)

#110

Post by tswartz »

chias wrote: post #105

I like the premise of this a lot, but it seems to me that the setting that the author chose (some UI element of a website) is one of the worst possible settings for this: what matters a whole lot more than if your button is red or green or blue is some modicum of consistency . If you're constantly changing the button color, size, location, whatever... that is an awful experience in and of itself, is it not? If the Am…

If you cookie the users and make sure the experience they saw is persistent that solves most of this problem. But if you run a lot of separate tests than it's hard to avoid this.
Post reply on HN