A side note:
I love the website UX/Design. Especially the flowing comments layout.
20 lines of code that beat A/B testing (2012)
41–50 of 164 posts
Re: 20 lines of code that beat A/B testing (2012)
#42Offtopic, but why do i have to enable Javascript to even see anything?
The same reason you have to have electricity to watch TV.
It's a required part of modern web sites -- and it doesn't matter whether it's "really needed" for any particular site or not.
Re: 20 lines of code that beat A/B testing (2012)
#43Earlier quoted context omitted.
That is really weird. Technically the content is there all along (so it's not loaded in by JavaScript) but you still have to have JavaScript enabled for it to render. Who designed that!? Edit: hahaha what. It appears the content is laid out with JavaScript. So basically they're using JavaScript as a more dynamic CSS. Let that sink in. They're using JavaScript as CSS. It sorta-kinda makes sense for the fancy stream of…
Apparently some sort of useless fade-in effect is more important than the page content. This is the fix: #main { visibility: visible !important; }
Their trick also keeps noscript weirdos out though, which is a plus.
Re: 20 lines of code that beat A/B testing (2012)
#44Earlier quoted context omitted.
Here are 2 good articles that follow up on the arguments presented by VWO in that article. From the first link below: "They do make a compelling case that A/B testing is superior to one particular not very good bandit algorithm, because that particular algorithm does not take into account statistical significance. However, there are bandit algorithms that account for statistical significance." * https://www.chrisstuc…
Chris is now VWO's director of data science. We recently overhauled our stats. Here's a quick summary for that: https://vwo.com/blog/smartstats-testing-for-truth/
Changing you users UI always bears some cost, though, so I'm not sure it's really the smart choice.
Re: 20 lines of code that beat A/B testing (2012)
#45"Like many techniques in machine learning, the simplest strategy is hard to beat." is a thoroughly ridiculous statement. It should instead say "Like many techniques in machine learning, the simplest strategy is easiest to implement" as the title of the post (20 lines) makes it clear.
Not just easiest to implement -- literally "hard to beat".
Not "impossible to beat" mind you -- merely hard. Because the simplest strategies work so well in so many cases, that you need to go quite subtle and implement harder algorithms (doesn't matter if you just take them ready off of a lib for the argument) to see an improvement. 80-20, et al.
Re: 20 lines of code that beat A/B testing (2012)
#46I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…
My professional advice to people who make these unmeasurable changes is: Then don't do them. The worst part about unmeasurable changes is you can't verify if it has a negative effect either and you're essentially saying "I'm focusing on making this change when it will have negligible impact on the business."
You have better things to do.
Re: 20 lines of code that beat A/B testing (2012)
#47Earlier quoted context omitted.
Apparently some sort of useless fade-in effect is more important than the page content. This is the fix: #main { visibility: visible !important; }
Their trick also keeps noscript weirdos out though, which is a plus.
No, it didn't. I'm not about to run random Javascript, and the page was visible within ~3 clicks.
Page that are actually broken without Javascript only indicate that the author was lazy and unprofessional. Progressive enhancement is not hard.
Re: 20 lines of code that beat A/B testing (2012)
#48Earlier quoted context omitted.
The points raised are valid, if they matter is a different beast Even in the tests shown, conversion rate was higher for the MABA algorithms than simple A/B testing. "Oh but you get higher statistical significance!" thanks, but that doesn't pay my bills, conversion pays.
Careful. It wasn't always higher for MAB even though the tables shown there make it appear so at first. Those tables are showing the conversion rate during the test, up to the time when statistical significance is achieved. You generally then stop the test and go with the winning option for all your traffic. In the two-way test where the two paths have real conversion rates of 10% and 20%, all of the MAB variations d…
> up to the time when statistical significance is achieved. You generally then stop the test
Just a note, don't literally do this:
http://conversionxl.com/statistical-significance-does-not-eq...
Re: 20 lines of code that beat A/B testing (2012)
#49Multivariate A/B testing lets you test combinations of multiple changes simultaneously.
Re: 20 lines of code that beat A/B testing (2012)
#50I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…
seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.
Let's assume your site is now working with option A. Conversion rate is 5.7%, measured over a month.
You're now running an A/B for 1 week with option A and another version, option B. You get a conversion rate of 6.5% for option A and 5.9% for option B. Normally, you'd say that 6.5% is better than 5.9%. But how sure can you be if you don't control the other factors and both are performing better than before? How many visitors do you need to offset the influence of offsite factors?