Lines of code that beat A/B testing (2012)
stevehanov.ca
Lines of code that beat A/B testing (2012)
1–10 of 180 posts
Re: Lines of code that beat A/B testing (2012)
#2Multi arm bandits are fine but their limited to tests where its ok to switch users between arms frequently and tests that have more power
Re: Lines of code that beat A/B testing (2012)
#3Re: Lines of code that beat A/B testing (2012)
#4"People distrust things that they do not understand, and they especially distrust machine learning algorithms, even if they are simple."
How times have changed :)
Re: Lines of code that beat A/B testing (2012)
#5As one of the comments below the article states, the probabilistic alternative to epsilon-greedy is worth exploring ad well. Take the "bayesian bandit", which is not much more complex but a lot more powerful.
If you crave more bandits: https://jamesrledoux.com/algorithms/bandit-algorithms-epsilo...
Re: Lines of code that beat A/B testing (2012)
#6Nothing shows on this page without JavaScript except for the header and a grey background. A bit strange for a blog.
Re: Lines of code that beat A/B testing (2012)
#7Nothing shows on this page without JavaScript except for the header and a grey background. A bit strange for a blog.
Utterly unremarkable for 2012 though.
Re: Lines of code that beat A/B testing (2012)
#8A lot of sites don't have enough traffic to get statistical significance with this in a reasonable amount of time and it's almost always testing a feature more complicated than button color where you aren't going to have more than the control and variant.
Re: Lines of code that beat A/B testing (2012)
#9As one of the comments below the article states, the probabilistic alternative to epsilon-greedy is worth exploring ad well. Take the "bayesian bandit", which is not much more complex but a lot more powerful. If you crave more bandits: https://jamesrledoux.com/algorithms/bandit-algorithms-epsilo...
We've been happy using Thompson sampling in production with this library https://github.com/bayesianbandits/bayesianbandits
Re: Lines of code that beat A/B testing (2012)
#10Multi arm bandits are fine but their limited to tests where its ok to switch users between arms frequently and tests that have more power
> where its ok to switch users between arms frequently
It's not hard to keep track of which arm any given user was exposed to in the first run, and then repeat it.