Interesting but since the writing of this article (2012) Google do offer Multi Armed Banned Content Experiments...
https://support.google.com/analytics/answer/2844870?hl=en
20 lines of code that beat A/B testing (2012)
21–30 of 164 posts
Re: 20 lines of code that beat A/B testing (2012)
#22I thought this was a pretty good follow up to show the strengths and weaknesses of this approach: https://vwo.com/blog/multi-armed-bandit-algorithm/ . Personally I think this approach makes a lot more sense than a/b testing especially when often people hand off the methodology to a 3rd party without knowing exactly how they work.
Here are 2 good articles that follow up on the arguments presented by VWO in that article. From the first link below: "They do make a compelling case that A/B testing is superior to one particular not very good bandit algorithm, because that particular algorithm does not take into account statistical significance. However, there are bandit algorithms that account for statistical significance." * https://www.chrisstuc…
Chris is now VWO's director of data science. We recently overhauled our stats. Here's a quick summary for that: https://vwo.com/blog/smartstats-testing-for-truth/
Re: 20 lines of code that beat A/B testing (2012)
#23Earlier quoted context omitted.
That is really weird. Technically the content is there all along (so it's not loaded in by JavaScript) but you still have to have JavaScript enabled for it to render. Who designed that!? Edit: hahaha what. It appears the content is laid out with JavaScript. So basically they're using JavaScript as a more dynamic CSS. Let that sink in. They're using JavaScript as CSS. It sorta-kinda makes sense for the fancy stream of…
Very funny. They have a div #main with visibility:hidden. Removing that rule from dev tools displays the full page without enabling JS. The comments block is a solved problem with CSS. Basically any grid layout toolkit does that as their very first demo.
Yeah, if you are going to do that just for a fade-in transition at least set it to hidden with an inline script block then someone with script turned off they get the content. You block rendering until everything is downloaded, but then so does your needless transition.
If you are worried about it breaking in some browsers because you are modifying a tag that has not yet had its content completed so isn't accessible in the DOM yet, or because you use a DOM manipulation library that hasn't loaded yet due to lazy loading, have the script add an extra wrapper instead of modifying the existing tag [i.e. document.write('';) directly after and document.write(''; before ]. Or, of course, just don't...
Re: 20 lines of code that beat A/B testing (2012)
#24"Like many techniques in machine learning, the simplest strategy is hard to beat." is a thoroughly ridiculous statement. It should instead say "Like many techniques in machine learning, the simplest strategy is easiest to implement" as the title of the post (20 lines) makes it clear.
That's not thoroughly ridiculous. For many problems in machine learning, k-nearest-neighbors and a large dataset is very hard to beat in terms of error rate. Of course, the time to run a query is beyond atrocious, so other models are favored even if kNN has a lower error rate.
[deleted]
Re: 20 lines of code that beat A/B testing (2012)
#25I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…
" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc."
I'd serve them at the same time, randomize which clients see which one.
Re: 20 lines of code that beat A/B testing (2012)
#2610% of the time, we choose a lever at random. The other 90% of the time, we choose the lever that has the highest expectation of rewards. There is a problem with strategies that change the distribution over time: Other factors change over time too. For example let's say over time the percentage of your traffic that comes from search engines increases. And this traffic converts better then your other traffic. And let'…
This is why you should segment traffic and run separate tests for each segment, whether you're using an A/B testing or a multi-armed banding algorithm.
Re: 20 lines of code that beat A/B testing (2012)
#27This is exactly what Adobe is already doing with its 1:1 product. But the fact is a 10% of traffic for a medium-size website won't get you the results you want quickly. 10% of traffic only works on enterprise-level websites.
Re: 20 lines of code that beat A/B testing (2012)
#28I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…
seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.
Re: 20 lines of code that beat A/B testing (2012)
#29I thought this was a pretty good follow up to show the strengths and weaknesses of this approach: https://vwo.com/blog/multi-armed-bandit-algorithm/ . Personally I think this approach makes a lot more sense than a/b testing especially when often people hand off the methodology to a 3rd party without knowing exactly how they work.
The points raised are valid, if they matter is a different beast Even in the tests shown, conversion rate was higher for the MABA algorithms than simple A/B testing. "Oh but you get higher statistical significance!" thanks, but that doesn't pay my bills, conversion pays.
Careful. It wasn't always higher for MAB even though the tables shown there make it appear so at first.
Those tables are showing the conversion rate during the test, up to the time when statistical significance is achieved. You generally then stop the test and go with the winning option for all your traffic.
In the two-way test where the two paths have real conversion rates of 10% and 20%, all of the MAB variations did win. Here is how many conversions there would be after 10000 visitors for that test, and how they compare to the A/B test:
RAND 1988
MAB-10 1997 +9
MAB-24 2001 +13
MAB-50 1996 +8
MAB-90 1994 +6
RAND 1987
MAB-10 1969 -18
MAB-50 1987 +0
MAB-77 1988 +1
(The third column in the above two tables remains the same if you change 10000 to something else, as long as that something else. MAB-10 beats RAND in the first test by 9 conversions, and loses by 18 conversions in the second test).
Re: 20 lines of code that beat A/B testing (2012)
#30John Langford and the team (Microsoft Research) have built a contextual bandit library in Vowpal Wabbit.
It can be used from active learning to changing webpage layouts to increase ad clicks. It has the best bounds out of all exploratory algorithms.
Structured contextual bandits that come with LOLS (another algorithm present in vowpal wabbit) is extremely powerful.
All for free under BSD3.