Wondering what everyone would tell someone new to a/b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!
Ask HN: What is your one tip about A/B testing you would share?
1–10 of 10 posts
Re: Ask HN: What is your one tip about A/B testing you would share?
#2Learn what "statistical power" (http://en.wikipedia.org/wiki/Statistical_power) means, and then understand that the fact n% of users prefer one option doesn't necessarily make it a good idea. Statistical significance is far more important than the actual result. This is a really good paper about it: http://www.qubitproducts.com/sites/default/files/pdf/most_wi...
Re: Ask HN: What is your one tip about A/B testing you would share?
#3That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version.
However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.
I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.
Re: Ask HN: What is your one tip about A/B testing you would share?
#4If you radically change a feature then regular users will first start playing with it. Simply because it's new. In that case your test needs be longer or exclude regular users.
Re: Ask HN: What is your one tip about A/B testing you would share?
#5http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... - to make clients 'stick' to upstreams, if you're using nginx.
Re: Ask HN: What is your one tip about A/B testing you would share?
#6Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.
Re: Ask HN: What is your one tip about A/B testing you would share?
#7Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right?
I have seen statistically significant differences in outcomes in A/A testing.
A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.
Re: Ask HN: What is your one tip about A/B testing you would share?
#8Ensure that your testing things that are actually going to impact your conversion rate or goals. You don't have to test everything - just the things that matter.
Re: Ask HN: What is your one tip about A/B testing you would share?
#9Kittens will always improve conversion rates.
Re: Ask HN: What is your one tip about A/B testing you would share?
#10Choose and test one independent variable at a time. Then, check your results for statistical significance.