Yep. I implemented this as a Java library a while ago (and other stuff): https://github.com/JensRantil/java-canary-tools Basically, I think feature flags and A/B tests aren't always needed to roll out experiments.
Lines of code that beat A/B testing (2012)
21–30 of 180 posts
Re: Lines of code that beat A/B testing (2012)
#22The "20" is missing from the title.
I believe hacker news automatically truncates the number at the beginning of titles
Re: Lines of code that beat A/B testing (2012)
#23If multi-arm bandits has piqued your curiosity, we recently added support for it to our feature flagging and experimentation platform, GrowthBook.
We talk about it here: https://blog.growthbook.io/introducing-multi-armed-bandits-i...
Re: Lines of code that beat A/B testing (2012)
#24No, multi-armed bandit doesn't "beat" A/B testing, nor does it beat it "every time". Statistical significance is statistical significance, end of story. If you want to show that option B is better than A, then you need to test B enough times. It doesn't matter if you test it half the time (in the simplest A/B) or 10% of the time (as suggested in the article). If you do it 10% of the time, it's just going to take you…
Multi-arm bandit does beat A/B testing in the sense that standard A/B testing does not seek to maximize reward during the testing period, MAB does. MAB also generalizes better to testing many things than A/B testing.
Re: Lines of code that beat A/B testing (2012)
#25Multi-armed bandits make a big assumption that effectiveness is static over time. What can happen is that if they tip traffic slightly towards option B at a time when effectiveness is higher (maybe a sale just started) B will start to overwhelmingly look like a winner and get locked in that state. You can solve this with propensity scores, but it is more complicated to implement and you need to log every interaction.
This objection is mentioned specifically in the post.
You can add a forgetting factor for older results.
Re: Lines of code that beat A/B testing (2012)
#26No, multi-armed bandit doesn't "beat" A/B testing, nor does it beat it "every time". Statistical significance is statistical significance, end of story. If you want to show that option B is better than A, then you need to test B enough times. It doesn't matter if you test it half the time (in the simplest A/B) or 10% of the time (as suggested in the article). If you do it 10% of the time, it's just going to take you…
Multi-arm bandit does beat A/B testing in the sense that standard A/B testing does not seek to maximize reward during the testing period, MAB does. MAB also generalizes better to testing many things than A/B testing.
No -- you can't have your cake and eat it too.
You get zero benefits from MAB over A/B if you simply end your A/B test once you've achieved statistical significance and pick the best option. Which is what any efficient A/B test does -- there no reason to have any fixed "testing period" beyond what is needed to achieve statistical significance.
While, to the contrary, the MAB described in the article does not maximize reward -- as I explained in my previous comment. Because the post's version runs indefinitely, it has worse long-term reward because it continues to test inferior options long after they've been proven worse. If you leave it running, you're harming yourself.
And I have no idea what you mean by MAB "generalizing" more. But it doesn't matter if it's worse to begin with.
(Also, it's a huge red flag that the post doesn't even mention statistical significance.)
Re: Lines of code that beat A/B testing (2012)
#27Here's an interesting write up on various algorithms & different epsilon greedy % values.
https://github.com/raffg/multi_armed_bandit
It shows 10% exploration performs the best, very great simple algorithm.
Also it shows the Thompson Sampling algorithm converges a bit faster-- the best arm chosen by sampling from the beta distribution, and eliminates the explore phase. And you can use the builtin random.betavariate !
https://github.com/raffg/multi_armed_bandit/blob/42b7377541c...
Re: Lines of code that beat A/B testing (2012)
#28> Statistics are hard for most people to understand.
True, but that's exactly what statistics helps with, though also hard to understand. :)
Re: Lines of code that beat A/B testing (2012)
#29Earlier quoted context omitted.
Multi-arm bandit does beat A/B testing in the sense that standard A/B testing does not seek to maximize reward during the testing period, MAB does. MAB also generalizes better to testing many things than A/B testing.
No -- you can't have your cake and eat it too. You get zero benefits from MAB over A/B if you simply end your A/B test once you've achieved statistical significance and pick the best option. Which is what any efficient A/B test does -- there no reason to have any fixed "testing period" beyond what is needed to achieve statistical significance. While, to the contrary, the MAB described in the article does not maximize…
> you can't have your cake and eat it too
I disagree. There is a vast array of literature on solving the MAB problem that may as well be grouped into a bin called “how to optimally strike a balance between having one’s cake and eating it too.”
The optimization techniques to solve MAB problem seek to optimize reward by giving the right balance of exploration and exploitation. In other words, these techniques attempt to determine the optimal way to strike a balance between exploring if another option is better and exploiting the option currently predicted to be best.
There is a strong reason this literature doesn’t start and end with: “just do A/B testing, there is no better approach”
Re: Lines of code that beat A/B testing (2012)
#30No, multi-armed bandit doesn't "beat" A/B testing, nor does it beat it "every time". Statistical significance is statistical significance, end of story. If you want to show that option B is better than A, then you need to test B enough times. It doesn't matter if you test it half the time (in the simplest A/B) or 10% of the time (as suggested in the article). If you do it 10% of the time, it's just going to take you…
[deleted]