One of the best talks I've ever seen was Aaron Swartz talking about how Victory Kit used multi-armed bandits to optimize political action: https://github.com/victorykit/whiplash
20 lines of code that beat A/B testing (2012)
141–150 of 164 posts
Re: 20 lines of code that beat A/B testing (2012)
#142Earlier quoted context omitted.
You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about t…
[Disclaimer: I previously worked for Optimizely as predictive analytics PM - but no longer work there, and don't speak for the company.]
Optimizely has a bandit based 'traffic auto-allocation' feature in production on select enterprise plans [1]; bandits are excellent in a wide range of situations, and have many advantages, but like anything, have design parameters and there are some caveats you have to be aware of to make sure you are using them effectively.
On Frequentist and Bayesian: Optimizely's stats engine combines elements of both Frequentist and Bayesian statistics. They have a blog that tries to touch on this issue [2] But this is subtle stuff - and there are a lot of trade-offs, and different perspectives; look at the Bayesian/frequentist debate which has been going on for decades among statisticians.
But, FWIW, I definitely saw Optimizely as an organisation make a big investment to produce a stats engine which had the right trade-offs for how their customers were trying to test; and I think the end result was way more suitable than 'traditional' statistics were.
[1] https://help.optimizely.com/hc/en-us/articles/200040115-Traf... "Traffic Auto-allocation automatically adjusts your traffic allocation over time to maximize the number of conversions for your primary goal. [...] To learn more about how algorithms like this work, you might want to read about a popular statistics problem called the “multi-armed bandit.”"
[2] https://blog.optimizely.com/2015/03/04/bayesian-vs-frequenti... "Yet as we developed a statistical model that would more accurately match how Optimizely’s customers use their experiment results to make decisions (Stats Engine), it became clear that the best solution would need to blend elements of both Frequentist and Bayesian methods to deliver both the reliability of Frequentist statistics and the speed and agility of Bayesian ones."
Re: 20 lines of code that beat A/B testing (2012)
#143"Like many techniques in machine learning, the simplest strategy is hard to beat." is a thoroughly ridiculous statement. It should instead say "Like many techniques in machine learning, the simplest strategy is easiest to implement" as the title of the post (20 lines) makes it clear.
That's not thoroughly ridiculous. For many problems in machine learning, k-nearest-neighbors and a large dataset is very hard to beat in terms of error rate. Of course, the time to run a query is beyond atrocious, so other models are favored even if kNN has a lower error rate.
According to [1], k-NN is pretty far from being the top general classifier. Admittedly, all the data sets are small, with no more than 130,000 instances. When does it start becoming "very hard to beat", and what are you basing that on?
1. http://jmlr.org/papers/volume15/delgado14a/delgado14a.pdf
Re: 20 lines of code that beat A/B testing (2012)
#144isn't this how a/b testing works with google analyitcs?
Re: 20 lines of code that beat A/B testing (2012)
#145Earlier quoted context omitted.
My professional advice to people who make these unmeasurable changes is: Then don't do them. The worst part about unmeasurable changes is you can't verify if it has a negative effect either and you're essentially saying "I'm focusing on making this change when it will have negligible impact on the business." You have better things to do.
I agree, every article about A/B testing is talking about making small changes and measure them, while you should be measuring only big changes or nothing at all.
[deleted]
Re: 20 lines of code that beat A/B testing (2012)
#146Earlier quoted context omitted.
The basic model that Optimizely uses is a Z-Test approximation of a binomial distribution. To run a proper experiment with that model, you should be calculating the sample size ahead of time, and then run it. Each visitor should be independent, and not affected by things like the day of the week, or the time of it. The end result tells you if the distributions are different, but not as much as one would think about t…
[Disclaimer: I previously worked for Optimizely as predictive analytics PM - but no longer work there, and don't speak for the company.] Optimizely has a bandit based 'traffic auto-allocation' feature in production on select enterprise plans [1]; bandits are excellent in a wide range of situations, and have many advantages, but like anything, have design parameters and there are some caveats you have to be aware of t…
Hi Fergal!
I didn't realize that the auto-allocation ever shipped, but I'm glad it finally did. Hopefully there was work done to empirically show that they solved a lot of the issues around time to convert and other messy parts of the data that killed earlier efforts, but I think everyone who knew about those was gone before you joined :)
There are very subtle issues with both frequentist and bayesian stats, which makes combining them sounds insane to me.
What are you up to these days?
Re: 20 lines of code that beat A/B testing (2012)
#147Earlier quoted context omitted.
You should contact Optimizely and let them know right now. https://www.optimizely.com/contact/ They'll need to reeducate their statisticians right away!
You're wrong. What you are thinking about is Optimizely's "Chance to Beat Baseline" number. That's different from the statistical significance, which is a setting you can change on the Settings page. Being smug and condescending really backfires when you don't know what you're talking about.
Bro... http://cl.ly/1b0B3Y3o1w09
> Being smug and condescending really backfires when you don't know what you're talking about.
How's that working out for you?
Re: 20 lines of code that beat A/B testing (2012)
#148Earlier quoted context omitted.
> but they never really published much justifying it Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...). I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course. [0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...
Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results. They are still having people make very fundamentally flawed assumptions about the data, which results in inco…
It may be marketing but I was able to implement a sequential A/B test based on it. Admittedly, I did need to do some work beyond merely copy/pasting an algorithm, but all I really needed to do was read their paper and some citations. I do believe that this document does describe a viable frequentist test and my implementation of it worked pretty well.
Disclaimer: I do stats work at VWO, an Optimizely competitor.
(Also if you want to read our tech paper, here it is: https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic... This describes our Bayesian approach, which we believe to be less likely to be wrongly interpreted by non-statisticians.)
Re: 20 lines of code that beat A/B testing (2012)
#149Earlier quoted context omitted.
>I am free to just spew out content regardless of its quality. Oh, that's a great goal to have...
By focusing on volume, I get quality as a side effect. I'm very proud of the the app I've created and the feedback I get from my users: "This app gives a text reminder to do what everyone wants to do: relax, love one's self and others, and bring peace and light into the world. I smile when the alert message appears and feel grateful to the makers of this app for creating a pleasant, quick way to meditate at the most…
Volume and efficiency are great, but nobody likes a shit cannon.
Re: 20 lines of code that beat A/B testing (2012)
#150Earlier quoted context omitted.
>I am free to just spew out content regardless of its quality. Oh, that's a great goal to have...
I actually think that's a great idea, because then the users and algorithm decide what's good for you, and it will hide the low quality content automatically. You have to do that sort of "curation" anyways anytime you make something. You have to continually decide whether it's worth it to keep working on something, and then decide if it's good enough to release. People tend to be pretty bad judges of this (especially…
"I promise, there is a diamond somewhere in this bathtub of shit".
Efficiency is great, but there still needs to be some quality standards.