11–20 of 80 posts
Reading the conclusions it seems that each approach - a/b testing vs multi-armed bandit - has its merits. Maybe Visual Website Optimizer should be method agnostic and offer customers the choice of method?
I would like to have seen more tests vs the MAB-90 algorithm as this appeared the most interesting one to me. Also the article appears to imply that A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument.
> A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument. Why do you say so? Can you elaborate?
In reality, however, tests like the Wilcoxin test are 99% as powerful and more robust to misspecification in models.
I bring this up because while statistical power and signficance is a very important metric in the theory of picking good tests, it's actually a pretty terrible one in practice. Comparing MAB, which optimizes an entirely different loss parameter, to t/z-tests on power is sort of meaningless.
MAB can produce a cleaner workflow for many kinds of websites. Underperforming classes will be underrepresented and eventually pruned. The increased power of a batch test isn't necessarily so important in this context. I'm not even actually advocating MAB over other tests, just that you shouldn't spend too much time worry about power comparisons unless you're genuinely comparing apples to apples.
From a statistics perspective, a few things are missing. Most importantly, a discussion of statistical power and a discussion of why exactly it is that they test until statistical difference is found. Every scientist knows that if you test a big enough sample, you are more than likely going to find a statistical difference, regardless of whether it actually exists. Hence, using only that as your heuristic for what ma…
( From reading the post quickly, I'm not left with the impression that the author understands the research they are 'refuting'. (Which I also only read quickly and cannot help understand.) This is just my impression .)
I would like to have seen more tests vs the MAB-90 algorithm as this appeared the most interesting one to me. Also the article appears to imply that A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument.
> A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument. Why do you say so? Can you elaborate?
The goal of A/B is to decide as quickly as possible for the best one.
The goal of bandit is to optimize given content.
I wouldn't use bandit to "decide" the button color but as a simple recommendation system. This seems by far more natural to me as it reacts better with changing optima.
Example: Let's say I run a larger "fun content" media webpage. I have "awesome videos", "funny images" and "goofy articles". On the bottom and right side of each content page i show follow up content. I would use bandit here to optimize the mix of images,videos,text i recommend.
To spice it up: I would create cohorts for my typical behaviour of users (e.g. registered male user) and only consider interactions of the last two weeks into my bandit calculations.