Most AB-tests will fail

Depends where the copy is. Changing a couple of words in an email subject line can and often does make a huge difference. Of course the fine line you walk is that some of the most effective words at driving behavior (Free! Now!) are also effective at getting you marked as spam.

In reverse order.

#3. For real companies with completely unmotivated visitors, 60% conversion rates from visitor to paying customer is untenable. 5% is probably too much to hope for.

#2. Absolutely. A/B tests should be focused on actual balanced decision points. "Should I sign up?" "Should I open this email?" "Should I go back to this site?" But not on people who are already committed to doing what they were going to do anyways.

#3. On sample sizes, it is important to think carefully about the maximum effort you're willing to put into a test. Be very, very cautious about accepting test results that arrive early. No, 99% confidence is not enough to stop with 200 conversions, and 99.9% probably isn't either. Don't worry about statistical significance when you kill tests that have run too long to be worth continuing. This is a line of reasoning that is sadly rare in our industry. (I keep meaning to write my next article on that topic. But http://elem.com/~btilly/ab-testing-multiple-looks/part2-limi... explains one way to come up with such a strategy.)

True, most A/B tests fail.

However looking at the listed tests, they mostly seem to be general branding tests. Changing your branding, including fairly major redesigns, reliably fails to make a difference.

But lots of other kinds of changes reliably do make a difference. Big button. Putting signup link in multiple places. The word "Free". Streamlining your form. (No, you don't want to ask for first/last name.) Email headlines. Providing relevant results first.

However they make small differences. You need to plan on tens of thousands of successful conversions in your test. You can make this easier by running multiple tests at once. If you're a small company, you can make conversion be "to the next step in the conversion funnel" instead of "to the end". (But be warned, companies that test long enough and get large enough tend to find examples of cases where that short cut burned them.)

However it is very important to set expectations. Expect to find few big wins. Expect most to be modest wins. Keep track. For an established company, if A/B testing is adding 5-20% to your bottom line each year, it is totally worthwhile. (If you have traffic and have not been testing, then odds are that your first year will find more than a 20% boost. But this declines in future years.)

A/B testing is a valuable tool. It isn't magic pixie dust.

Couple points there:

#1 The article is very vague - the Obama team may have actually made a very common mistake and stopped the tests early when they hit a significant level (they were in a big rush at the time, they had days to get everything working, not months). We don't know anything about samples sizes, etc.

#2 The main detail they talk about changing was changing from a movie to a still picture. This is a lot different than changing a button's color - many people will see a movie and close the page if they are in a quiet environment such as work or a library.

So it could very well contradict the A/B testing article by Wired and still be correct - or it could be wrong. Contradiction itself is not a red flag here in something as new as A/B web testing.

Thanls a lot for taking your time to write this awesome comment!

The thing is we have very little control over the checkout area, since it is hosted by our payment provider and we cannot run tests there. We are not happy about it and we will move to Stripe as soon as it will be available outside of the US.

Really, thnaks a lot for this. I will look into this right now, maybe we could fix some of those.

EDIT: We've removed the quantity field from the SaaS products, never thought of that before, thanks.