[deleted]
The multi-armed bandit problem (2012)
21–30 of 82 posts
Re: The multi-armed bandit problem (2012)
#22Earlier quoted context omitted.
Yes. Bandits will often converge more quickly to the optimal strategy, but it is much more difficult to understand why that strategy is optimal and generalize from the bandit outcomes to predict future performance and performance of other strategies. It isn't impossible - bandits are seeing adoption in medical trials to avoid precisely the problem discussed - but the standard experiment design and analysis techniques…
I disagree. I’ve spent a lot of time staring at bandit outcomes and usually they match some sort of intuition of why a variant might be exceptional.
That could be post-hoc reasoning, though. It would be interesting to pre-register your hypotheses, or see whether you could tell bandit outcomes from random ones.
Re: The multi-armed bandit problem (2012)
#23The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from which you can learn things. Also, I work in this field and I will just say that people _do_ behave differently based on traffic source: i.e. users coming from Facebook behave alike, but different than traffic from Reddit who act similarly to each other. If you wer…
> The purpose of an A/B test isn't to always show the best performing result
The key part is 'not always'.
Using A/B tests just to find the better performing version is a valid use case too. And if you have the traffic, multi-armed bandits are a nice way of automating the whole procedure. Even scientifically speaking there is nothing wrong with them. Their biggest issue is that they require a lot of traffic for significant results.
Re: The multi-armed bandit problem (2012)
#24Also one reason a lot of teams can't do more than two options (A, B, C, D, E, F, G, etc. testing) is because you need a TON of traffic for it to be statistically significant.
Statistical significance isn't necessary for deriving value from information! The point of multi-armed bandit is that you use the best information you currently have, while also not taking that information too seriously. In a context where experiments have a cost and you need results, this makes more sense than gathering more and more data until you meet statistical significance thresholds.
Re: The multi-armed bandit problem (2012)
#25The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from which you can learn things. Also, I work in this field and I will just say that people _do_ behave differently based on traffic source: i.e. users coming from Facebook behave alike, but different than traffic from Reddit who act similarly to each other. If you wer…
What you said is correct, but I'd like to point out that controlled scientific experiments may not be the right approach for e.g. optimizing conversions on a website. The reason is that websites are a dynamic environment. All things equal, better controlled experiments are great. However, visitor behavior, especially when from dynamic sources (google serps change weekly), changes all the time. And that's why I prefer…
> Does anyone go back and re-test their biggest wins?
Yes, absolutely! We do research first, then come up with simple, well-controlled tests. Once we have a winner we can either lock it in, but often we continue to research and experiment on the new knowledge we gained. A hefty minority of the tests I implement build on past wins to further flesh out what works and what doesn't with knowledge and the data to back it up.
Re: The multi-armed bandit problem (2012)
#26Earlier quoted context omitted.
The user id can be used to seed the random number algorithm to achieve that.
Hmmm but if you change the distribution, this won't be deterministic anymore unless the randomness only applies to new users entering the test setup.
Which is necessary. You don’t want to compare two wildly different users. This is why you need to define criteria for eligibility.
Re: The multi-armed bandit problem (2012)
#27One thing that is vastly more expensive than picking a winner in your A/B test is maintaining N variations across M features.
Re: The multi-armed bandit problem (2012)
#28The purpose of an A/B test isn't to always show the best performing result, it's to perform a _controlled scientific experiment_ with a control group, from which you can learn things. Also, I work in this field and I will just say that people _do_ behave differently based on traffic source: i.e. users coming from Facebook behave alike, but different than traffic from Reddit who act similarly to each other. If you wer…
That’s just a contextual multi-armed bandit.
Re: The multi-armed bandit problem (2012)
#29Just a small nitpick: this doesn't take into account implementation cost. If you want something dynamic like this it means your app has read access to all the analytics recorded (or at least the ones needed for optimization). Most of the times apps only send data to the analytics services, developers read/analyze them and act based on the data. I personally didn't work on any apps that were using analytics read access within the app. So, in most cases it's a lot more than 20 lines to implement an approach like this (hopefully your analytics platform exposes an API with a read endpoint that you have to then integrate into your app) compared to A/B testing, where you just show several versions and then analyze the data and iterate.
Re: The multi-armed bandit problem (2012)
#30UCB1 is really not that much more complicated than epsilon-greedy. Some slightly sloppy code I wrote a few years ago, maybe 20 lines of code: https://github.com/j2kun/ucb1/blob/master/ucb1.py#L7-L35
Sure you have to read a bit more to know why it works, but if you write your code well you could plug this in without any extra trouble. It's not like you need a special optimization solver as a dependency.