Offtopic, but why do i have to enable Javascript to even see anything?
In Firefox, "View", "Page Style", "No Style". I assume other browsers let you do the same thing. This works for many pages that fail to render without JavaScript.
20 lines of code that beat A/B testing (2012)
11–20 of 164 posts
Re: 20 lines of code that beat A/B testing (2012)
#12 10% of the time, we choose a lever at random. The
other 90% of the time, we choose the lever that has
the highest expectation of rewards.
For example let's say over time the percentage of your traffic that comes from search engines increases. And this traffic converts better then your other traffic. And let's say at the same time, your yellow button gets picked more often then your green button.
This will make it look like the yellow button performs better then it actually does. Because it got more views during a time where the traffic was better.
This can drive your website in the wrong direction. If the yellow button performs better at first just by chance then it will be displayed more and more. If at the same time the quality of your traffic improves, that makes it look like the yellow button is better. While in reality it might be worse.
In the end, the results of these kinds of adaptive strategies are almost impossible to interpret.
Re: 20 lines of code that beat A/B testing (2012)
#13I thought this was a pretty good follow up to show the strengths and weaknesses of this approach: https://vwo.com/blog/multi-armed-bandit-algorithm/ . Personally I think this approach makes a lot more sense than a/b testing especially when often people hand off the methodology to a 3rd party without knowing exactly how they work.
Here are 2 good articles that follow up on the arguments presented by VWO in that article.
From the first link below: "They do make a compelling case that A/B testing is superior to one particular not very good bandit algorithm, because that particular algorithm does not take into account statistical significance.
However, there are bandit algorithms that account for statistical significance."
* https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs...
* https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm...
Re: 20 lines of code that beat A/B testing (2012)
#14I thought this was a pretty good follow up to show the strengths and weaknesses of this approach: https://vwo.com/blog/multi-armed-bandit-algorithm/ . Personally I think this approach makes a lot more sense than a/b testing especially when often people hand off the methodology to a 3rd party without knowing exactly how they work.
The points raised are valid, if they matter is a different beast
Even in the tests shown, conversion rate was higher for the MABA algorithms than simple A/B testing. "Oh but you get higher statistical significance!" thanks, but that doesn't pay my bills, conversion pays.
Re: 20 lines of code that beat A/B testing (2012)
#1510% of the time, we choose a lever at random. The other 90% of the time, we choose the lever that has the highest expectation of rewards. There is a problem with strategies that change the distribution over time: Other factors change over time too. For example let's say over time the percentage of your traffic that comes from search engines increases. And this traffic converts better then your other traffic. And let'…
I don't know if this is the case if I understand this algorithm correctly. Say Yellow is 50%, and Green is a 65% success rate after the behavior change, but green is 30% before the behavior change.
By sending 90% of traffic towards yellow, it's ratio will normalize towards the 50% once it has enough traffic. By sending 10% of traffic randomly, eventually the green option will reach 51%, and start taking a majority of traffic, which then will cause it to normalize at it's 65%, and be shown to a majority of users.
I think the problem might be, if you run this with a sufficiently high volume or for a long period of time, that if a behaviour change takes place it will take a long time to learn the new behaviour. Or if two options aren't actually different, it may continually flip back and fourth between two options.
Also, to me, the concept of A/B testing certain things may also have an undesired consequence. For example, I order from amazon every day, but today the but button is blue, what does that actually mean? And I go back to the site later and it's yellow again. There are still many people who get confused by seemingly innocuous changes with the way their computer interacts with them.
Re: 20 lines of code that beat A/B testing (2012)
#16"Like many techniques in machine learning, the simplest strategy is hard to beat." is a thoroughly ridiculous statement. It should instead say "Like many techniques in machine learning, the simplest strategy is easiest to implement" as the title of the post (20 lines) makes it clear.
That's not thoroughly ridiculous.
For many problems in machine learning, k-nearest-neighbors and a large dataset is very hard to beat in terms of error rate. Of course, the time to run a query is beyond atrocious, so other models are favored even if kNN has a lower error rate.
Re: 20 lines of code that beat A/B testing (2012)
#17I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird.
For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better.
But how many websites / apps really have >100K uniques? If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc.
And if you have In my experience, only when you test some completely different pages you'll see some significant differences.
Re: 20 lines of code that beat A/B testing (2012)
#18there is a startup (unlaunched) built around this approach http://www.growthgiant.com
Re: 20 lines of code that beat A/B testing (2012)
#19A/B Tasty (most used platform in France I think, not affiliated) uses this approach :
http://blog.abtasty.com/en/clever-stats-finally-statistics-s...
Re: 20 lines of code that beat A/B testing (2012)
#20Earlier discussion (989 points, 1407 days ago) https://news.ycombinator.com/item?id=4040022
I really wish hackernews would change the date format to days, months and years, rather than just days, or the original submission date as the title attribute of "1407 days ago".
Submission date is may 30, 2012