Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

71–80 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#71

Post by MJSplot_author »

rco8786 wrote: post #63
mangeletti wrote: post #60

Earlier quoted context omitted.

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll f…

You really should question the statistical significance of those numbers.

Running their numbers:

Assuming the usual sqrt(n+1) error on a count, as n is low. Combining the uncertainties from 83/66 [0] gives 1.258 +- 0.231 (1sd). It's only just likely that it is an actual improvement, but there is about a 30% chance that the orignal was better.

[0] (((1+66^0.5)/66)^2+((1+83^0.5)/83)^2)^0.5x83.0/66

EDIT

If we also include the errors on the total counts and use the fraction 83.0/66/6362x6392, then the error is:

(((1+66^0.5)/66)^2+((1+83^0.5)/83)^2+((1+6362^0.5)/6362)^2+((1+6392^0.5)/6392)^2)^0.5x83.0/66/6362x6392

which shows an improvement of 1.264+-0.234. Stastically nout.

Re: 20 lines of code that beat A/B testing (2012)

#73

Post by TeMPOraL »

coldtea wrote: post #69
pdkl95 wrote: post #47

Earlier quoted context omitted.

No, it didn't. I'm not about to run random Javascript, and the page was visible within ~3 clicks. Page that are actually broken without Javascript only indicate that the author was lazy and unprofessional. Progressive enhancement is not hard.

> Page that are actually broken without Javascript only indicate that the author was lazy and unprofessional. Or that they have kept with the times and understand the notion of opportunity cost. > Progressive enhancement is not hard. No, just redundant and for marginal benefit. The train has long left that station. Search Engines (well, those that matter) and screen readers for a11y both work with JS.

> Or that they have kept with the times and understand the notion of opportunity cost.

That's not keeping up with times, that's poisoning the well because others do it too. By making such choices a site author is making the Internet worse for everyone.

I'm starting to think there should be some kind of "how not to be an ass on the Internet" course that would be mandatory before you're allowed to do business on-line.

Re: 20 lines of code that beat A/B testing (2012)

#75

Post by MJSplot_author »

rco8786 wrote: post #63
mangeletti wrote: post #60

Earlier quoted context omitted.

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll f…

You really should question the statistical significance of those numbers.

The fact that on the summary page there isn't a number showing the significance is very worrying. As the whole reason for it is to make a binary choice, there could be a 2 line chuck of js that just says "B might be better (60% probablity)".

Re: 20 lines of code that beat A/B testing (2012)

#76

Post by orasis »

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels.

My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content.

By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just create.

There is an interesting study featured in "Thinking Fast and Slow" where they had two groups in a pottery class. The first group would have the entirety of their grade based on the creativity of a single piece they submit. The second group was graded on only the total number of pounds of clay they threw.

The second group crushed the first group in terms of creativity.

Re: 20 lines of code that beat A/B testing (2012)

#77

Post by kevin_nisbet »

TuringTest wrote: post #32
kevin_nisbet wrote: post #15

Earlier quoted context omitted.

I don't know if this is the case if I understand this algorithm correctly. Say Yellow is 50%, and Green is a 65% success rate after the behavior change, but green is 30% before the behavior change. By sending 90% of traffic towards yellow, it's ratio will normalize towards the 50% once it has enough traffic. By sending 10% of traffic randomly, eventually the green option will reach 51%, and start taking a majority of…

> Also, to me, the concept of A/B testing certain things may also have an undesired consequence. For example, I order from amazon every day, but today the but button is blue, what does that actually mean? And I go back to the site later and it's yellow again. There are still many people who get confused by seemingly innocuous changes with the way their computer interacts with them. Proper A/B tests are supposed to be…

I agree. Sorry in this context I was referring specifically to the "in 20 lines of code article" which I don't believe had this control to it.

Re: 20 lines of code that beat A/B testing (2012)

#78

Post by whatever_dude »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

Love that example, even if frequently retold.

http://blog.codinghorror.com/quantity-always-trumps-quality/

Re: 20 lines of code that beat A/B testing (2012)

#79

Post by [deleted] »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

[deleted]

Re: 20 lines of code that beat A/B testing (2012)

#80

Post by AJ007 »

spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

You are right, and the guy that responded with a test of just under 150 conversions is a great example of exactly what people get wrong.

I have been doing design optimization for nearly a decade now and virtually every example I've seen made public is incredibly poor. Sometimes they do not even include any numbers, but just say, look first I had a 1% conversion rate and now I got 1.2%! I wish I had a good example of optimization case study, but I don't recall the last time I saw one.

Without giving anything away, I have done tests this year which required over a million impressions to find out in a statistically significant way not to make any changes.

My general theory is that a never optimized design with a confusing UI has a lot of low hanging fruit. You start cleaning up the bad elements and conversion rates can double or triple. Even if the sample size is less than ideal, these really big pops will be apparent.

Design knowledge is better now than it was in 2003. Mobile forces designers to use one column and leave out a lot of crap. There are a lot of good examples that get copied and good out of the box software. That means when you start optimizing, the low hanging fruit is gone and you need really big sample sizes. Once the low hanging fruit is gone, often those big samples just tell you not to make any changes.

Thinking about free-to-play mobile games recently (an area I have no experience in.) If 1% or fewer of users are converting you really do need a huge install base to beat your competitors at optimization. You need millions of users just to get to 20,000 or 30,000 paying players to test behavioral changes on. That means there actually is some staying power for the winners, at least for a while.

Post reply on HN