Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

131–140 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#131

Post by blakeyrat »

butler14 wrote: post #28
spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.

I think the goal with this new mechanism is having the script "automatically" promote your Christmas-themed creatives in December, and go back to your normal creatives in January.

It seems like you could "bucket" results by week, to get some kind of date factor considered. Of course it's also possible your Christmas creatives are simply garbage and will never get promoted, even when they are in-season. ;)

Re: 20 lines of code that beat A/B testing (2012)

#132

Post by blakeyrat »

chias wrote: post #105

I like the premise of this a lot, but it seems to me that the setting that the author chose (some UI element of a website) is one of the worst possible settings for this: what matters a whole lot more than if your button is red or green or blue is some modicum of consistency . If you're constantly changing the button color, size, location, whatever... that is an awful experience in and of itself, is it not? If the Am…

One aspect of testing they leave unsaid is you identify your users (cookie, most commonly) to make sure each user always gets the same experience. That's why your numbers are all based on unique users, not merely users.

Their experience will still change once their cookie expires, but that amount of time is completely under your control.

Re: 20 lines of code that beat A/B testing (2012)

#133

Post by scottlocklin »

gburt wrote: post #125
scottlocklin wrote: post #123

Reinforcement approaches are certainly interesting, but one of the things missing here (and in most A/B stuff) is statistical significance and experimental power. If you have enough data, there are hand wavey arguments that this will eventually be right, but in the meanwhile, if there is some opportunity cost (say, imagine this is a trading algo trying to profit from bid/ask), you screwed yourself out of some unknown…

Any chance you could point me to a reference? I'm doing research in this space and currently working on a paper which does exactly this for diagnostics of testing processes.

I'm not sure which thing I mentioned you need a reference on. For p-values on machine learning techniques, http://vovk.net/cp/index.html I'll eventually do a blog post on this subject; it's very good math that all ML people should know about, though Vovk, Schaefer and Gammerman write pretty dense articles.

For statistical power... "Statistical Power Analysis for the Behavioral Sciences" by Jacob Cohen.

Re: 20 lines of code that beat A/B testing (2012)

#134

Post by gburt »

scottlocklin wrote: post #133
gburt wrote: post #125

Earlier quoted context omitted.

Any chance you could point me to a reference? I'm doing research in this space and currently working on a paper which does exactly this for diagnostics of testing processes.

I'm not sure which thing I mentioned you need a reference on. For p-values on machine learning techniques, http://vovk.net/cp/index.html I'll eventually do a blog post on this subject; it's very good math that all ML people should know about, though Vovk, Schaefer and Gammerman write pretty dense articles. For statistical power... "Statistical Power Analysis for the Behavioral Sciences" by Jacob Cohen.

Sorry, I intended to quote your last sentence, applying a confidence interval to a reinforcement learning system, especially with respect to multi-armed bandits / adaptive experiments, but if I have to dig in to some signal processing stuff I am happy to do that.

It seems the conformal prediction link has some relation. I will dig, thanks.

Re: 20 lines of code that beat A/B testing (2012)

#135

Post by norkakn »

gingerlime wrote: post #128
norkakn wrote: post #106

Earlier quoted context omitted.

The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about t…

> but they never really published much justifying it Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...). I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course. [0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...

Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results.

They are still having people make very fundamentally flawed assumptions about the data, which results in incorrect conclusions, and they are still not presenting the results in a way that people correctly interpret them. That being said, those are really hard to solve, and models that would try to correct for them would likely require a lot more data and be overly conservative for more people.

What are your reasons for disliking Optimizely?

Re: 20 lines of code that beat A/B testing (2012)

#136

Post by fapjacks »

orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

There were two epiphanies hidden in this comment. HN is good for one or two every once in a while, but for me personally, this struck gold. Thanks!!

Re: 20 lines of code that beat A/B testing (2012)

#138

Post by ced »

mangeletti wrote: post #117
ced wrote: post #94

Earlier quoted context omitted.

I'll add my own Bayesian analysis to the fray. Assuming a binomial, in Julia: using Distributions b_old = Beta(66+1, 6392-66+1) b_yel = Beta(83+1, 6362-83+1) N = 1000000 # Sample from both distributions, count the fraction of samples that are better sum(rand(b_old, N) .> rand(b_yel, N)) / N This yields 7.7% chance that the old one is better than "yellow". It's fascinating to see how we can get such different answers…

Thanks for this analysis. I did an A/B test on an older framework which didn't automate statistical significance at all, but the website was getting more than 2000-3000 orders per day, so after a single week we had enough data to determine that sales had increased by 36% (reduced the page's load time by almost half, changed the checkout to use Ajax, and a few other small changes.) without the need to quantify things.…

Good point, there are always unmodeled factors and prior information, so the % is to be interpreted in context.

Re: 20 lines of code that beat A/B testing (2012)

#139

Post by areyousure »

kens wrote: post #101
orasis wrote: post #76

Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just…

I tried to find the original source of the quality-vs-quantity pottery class story a while back. I think it originates in the book "Art and Fear" but in that book it reads like a parable rather than a factual event. I'm highly suspicious of whether this event actually happened. Anyone have solid evidence?

Having researched this myself moderately extensively, I do not believe that the event actually happened.

Moreover, the book "Thinking fast and slow" does not contain the word "pottery" (nor "ceramics").

Re: 20 lines of code that beat A/B testing (2012)

#140

Post by gk1 »

mangeletti wrote: post #93
adevine wrote: post #86

Earlier quoted context omitted.

> I wanted Optimizely to say it was 100% significant That's not how statistical significance works...

You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!

You're wrong. What you are thinking about is Optimizely's "Chance to Beat Baseline" number. That's different from the statistical significance, which is a setting you can change on the Settings page.

Being smug and condescending really backfires when you don't know what you're talking about.

Post reply on HN