Most AB-tests will fail
jitbit.com
Most AB-tests will fail
1–10 of 43 posts
Re: Most AB-tests will fail
#2> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it.
This contradicts the article about A/B testing by Wired, which used the Obama campaign as an example.
Re: Most AB-tests will fail
#3If all you test is the layout then, for the most part, you are Right! Right! Right!
But...
Even with little traffic, you can still achieve significant gains by focusing on the copy as opposed to the layout and colours.
I learnt this mostly with Adwords. At the prices they charge now, you just have to find the gains wherever they are.
Re: Most AB-tests will fail
#4Using A/B testing and then measuring conversion rate skips a lot of useful steps.
A/B testing can be used to increase engagement with the site, which may result in a conversion days or weeks later.
Re: Most AB-tests will fail
#5"No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it."
This is not entirely true, if you have enough traffic, even the slightest change can make a difference. HOWEVER, for 99% of sites what you say is true. A/B testing should be used last, to squeeze out the last few percents of optimization, once you've done everything else. Not as a 'first step'.
One cool thing that can come of these tests, as an anecdote, is to see what is really working on a page, as opposed to what needs to be optimized.
We ran a Multivariate Test once for a signup form. The results were all similar, except for 1 which was much, much worse then the others. We could not understand why because there were no big differences between the elements, but this one particular combination set was having a bad time at it. Turns out, the headline was longer for one of the variations. This wrapped the text and pushed the call to action down below the fold, resulting in a ~30% (negative) difference in the conversion from the ones that were above the fold.
Re: Most AB-tests will fail
#6"No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it." This is not entirely true, if you have enough traffic, even the slightest change can make a difference. HOWEVER, for 99% of sites what you say is true. A/B testing should be used last, to squeeze out the last few percents of optimization, once you've done everything else. Not as a 'first step…
Those things happened (and still happen) a lot to us. At first we were all excited when there was a significant difference between alternatives, but now I just assume that something is wrong by default.
Re: Most AB-tests will fail
#7If all you test is the layout then, for the most part, you are Right! Right! Right! But... Even with little traffic, you can still achieve significant gains by focusing on the copy as opposed to the layout and colours. I learnt this mostly with Adwords. At the prices they charge now, you just have to find the gains wherever they are.
Same principle applies to copy, especially to microcopy. If you change a couple of words in a headline, it rarely makes a difference.
Re: Most AB-tests will fail
#8Using A/B testing and then measuring conversion rate skips a lot of useful steps. A/B testing can be used to increase engagement with the site, which may result in a conversion days or weeks later.
Yes, but those things are impossible to track. If metrics changed weeks later, there is no way of knowing that it was caused by a test you run earlier.
Re: Most AB-tests will fail
#9The fact that 9 out of 10 tests don't impact the conversion rate is not a failure, it's just a warning that you're spendin g too much time on design, trying out stuff that people don't really care about. That said, having 1 test out of 10 giving you significant knowledge about which version increases or decreases the conversion rate is definitely worth the 9 others that weren't conclusive!
Re: Most AB-tests will fail
#10> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it. This contradicts the article about A/B testing by Wired, which used the Obama campaign as an example. http://www.wired.com/business/2012/04/ff_abtesting
Alright, that was a bold statement on my side. That's true for 99.9% of the companies, not 100%.
Again, most of us don't have enough traffic to make successful buton color tests.