Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

121–130 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#121

Post by SonicSoul »

orasis wrote: post #111
kens wrote: post #101

Earlier quoted context omitted.

I tried to find the original source of the quality-vs-quantity pottery class story a while back. I think it originates in the book "Art and Fear" but in that book it reads like a parable rather than a factual event. I'm highly suspicious of whether this event actually happened. Anyone have solid evidence?

It was featured in "Thinking Fast and Slow", and those authors seem quite academically rigorous.

that's an appeal to authority, sounds like @kens actually tried to verify this story. I'd be curious to hear more examples as well. I believe I heard something similar on You're Not So Smart podcast but that might have referenced the same example

Re: 20 lines of code that beat A/B testing (2012)

#122

Post by tswartz »

orasis wrote: post #120
tswartz wrote: post #108

Earlier quoted context omitted.

Good point! What do you use to do in-app A/B testing?

I implemented my own that I'll be making available for others soon at improve.ai (the site isn't up yet)

cool! good luck. let me know what it is when it launches. I'm in the market for a good iOS sdk. @tylerswartz

Re: 20 lines of code that beat A/B testing (2012)

#123

Post by scottlocklin »

Reinforcement approaches are certainly interesting, but one of the things missing here (and in most A/B stuff) is statistical significance and experimental power. If you have enough data, there are hand wavey arguments that this will eventually be right, but in the meanwhile, if there is some opportunity cost (say, imagine this is a trading algo trying to profit from bid/ask), you screwed yourself out of some unknown amount of profits. There actually ways of hanging a confidence interval on this approach which virtually nobody outside the signal processing and information theory communities know about. Kind of a shame.

Re: 20 lines of code that beat A/B testing (2012)

#124

Post by soperj »

SonicSoul wrote: post #121
orasis wrote: post #111

Earlier quoted context omitted.

It was featured in "Thinking Fast and Slow", and those authors seem quite academically rigorous.

that's an appeal to authority, sounds like @kens actually tried to verify this story. I'd be curious to hear more examples as well. I believe I heard something similar on You're Not So Smart podcast but that might have referenced the same example

I do think the example is bullshit, simply because you could throw ONE very thick pot and have it weight more than 50 pounds. Of course it's a horrible pot and you wouldn't get any better since you only made one piece, but it would get you an A since it would use more clay. Total # of pieces thrown would make more sense than pounds of clay used.

Re: 20 lines of code that beat A/B testing (2012)

#125

Post by gburt »

scottlocklin wrote: post #123

Reinforcement approaches are certainly interesting, but one of the things missing here (and in most A/B stuff) is statistical significance and experimental power. If you have enough data, there are hand wavey arguments that this will eventually be right, but in the meanwhile, if there is some opportunity cost (say, imagine this is a trading algo trying to profit from bid/ask), you screwed yourself out of some unknown…

Any chance you could point me to a reference? I'm doing research in this space and currently working on a paper which does exactly this for diagnostics of testing processes.

Re: 20 lines of code that beat A/B testing (2012)

#126

Post by SonicSoul »

kens wrote: post #101
orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I tried to find the original source of the quality-vs-quantity pottery class story a while back. I think it originates in the book "Art and Fear" but in that book it reads like a parable rather than a factual event. I'm highly suspicious of whether this event actually happened. Anyone have solid evidence?

yeah this is referenced all over the place (Derek Sivers, Jeft Atwood, Kevin Kelly) but always just this one paragraph and comes from this book http://kk.org/cooltools/art-fear/ don't see any references there either.

Re: 20 lines of code that beat A/B testing (2012)

#127

Post by disgruntledphd2 »

hythloday wrote: post #48
tzs wrote: post #29

Earlier quoted context omitted.

Careful. It wasn't always higher for MAB even though the tables shown there make it appear so at first. Those tables are showing the conversion rate during the test, up to the time when statistical significance is achieved. You generally then stop the test and go with the winning option for all your traffic. In the two-way test where the two paths have real conversion rates of 10% and 20%, all of the MAB variations d…

> up to the time when statistical significance is achieved. You generally then stop the test Just a note, don't literally do this: http://conversionxl.com/statistical-significance-does-not-eq...

Just to reiterate, this violates the assumptions under which you get your p-values.

I want an A/B testing tool that won't let you see results until it's done.

Re: 20 lines of code that beat A/B testing (2012)

#128

Post by gingerlime »

norkakn wrote: post #106
mangeletti wrote: post #93

Earlier quoted context omitted.

You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!

The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about t…

> but they never really published much justifying it

Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...).

I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course.

[0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...

Re: 20 lines of code that beat A/B testing (2012)

#129

Post by hinkley »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

You might want to be careful with your conclusions.

We don't know from this study whether the second group was more or less likely to get trapped in the Expert Beginner phase of development.

You definitely don't get anywhere without practice, but you are likely to get nowhere fast without theory.

Re: 20 lines of code that beat A/B testing (2012)

#130

Post by wbeckler »

ced wrote: post #94
rco8786 wrote: post #63

Earlier quoted context omitted.

You really should question the statistical significance of those numbers.

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers…

I get about the same result using my favorite online Bayesian split test calculator: http://www.peakconversion.com/2012/02/ab-split-test-graphica...
Post reply on HN