Most AB-tests will fail • Hacker News Board

Re: Most AB-tests will fail

#11

Post by mjhea0 » Wed, Jun 19, 2013, 2:37 PM UTC

m4xt3r wrote: post #10

Slix wrote: post #2

> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it. This contradicts the article about A/B testing by Wired, which used the Obama campaign as an example. http://www.wired.com/business/2012/04/ff_abtesting

Alright, that was a bold statement on my side. That's true for 99.9% of the companies, not 100%. Again, most of us don't have enough traffic to make successful buton color tests.

thank you!

i cannot tell you how excited i get when a client wants to split only to change a color of a button or a subtle change in a particular font. so frustrating!

Re: Most AB-tests will fail

#12

Post by ebbv » Wed, Jun 19, 2013, 2:49 PM UTC

In our bizdev meeting yesterday we had a conversation that was very much along the lines of this post. It mirrors our experience over years of A/B testing completely.

Re: Most AB-tests will fail

#13

Post by posabsolute » Wed, Jun 19, 2013, 2:50 PM UTC

Not quite sure I fully agree with this, having played myself with a/b tests for big box publicity, changing the message as lead to tremendous changes. Same for the website, conversions has changed a lot, not always for the better.

I do fully agree that buttons color really doesn't matter, which may explain OP new vs old website conversion rate, frankly it's about the same layout, without any different call to actions.

Re: Most AB-tests will fail

#14

Post by TallboyOne » Wed, Jun 19, 2013, 2:51 PM UTC

This article is full of so much wrong I can't even finish reading it. Why is this on the first page.

Source: I do conversion optimization for a living.

Re: Most AB-tests will fail

#15

Post by lucisferre » Wed, Jun 19, 2013, 2:52 PM UTC

TallboyOne wrote: post #14

This article is full of so much wrong I can't even finish reading it. Why is this on the first page. Source: I do conversion optimization for a living.

Source: Your opinion. Perhaps you would like to elaborate and add to the conversation instead.

Re: Most AB-tests will fail

#16

Post by posabsolute » Wed, Jun 19, 2013, 2:55 PM UTC

TallboyOne wrote: post #14

This article is full of so much wrong I can't even finish reading it. Why is this on the first page. Source: I do conversion optimization for a living.

Yeah..

If you don't have anything to give to the discussion, maybe you should not comment.

Re: Most AB-tests will fail

#17

Post by lucisferre » Wed, Jun 19, 2013, 2:55 PM UTC

m4xt3r wrote: post #8

grandalf wrote: post #4

Using A/B testing and then measuring conversion rate skips a lot of useful steps. A/B testing can be used to increase engagement with the site, which may result in a conversion days or weeks later.

Yes, but those things are impossible to track. If metrics changed weeks later, there is no way of knowing that it was caused by a test you run earlier.

It's easier and easier to track, ad retargeting works this way for example. But yes it is still harder to track accurately.

Re: Most AB-tests will fail

#18

Post by AJ007 » Wed, Jun 19, 2013, 3:03 PM UTC

#1 Its good to see a post using statistically significant sample sizes. Most articles on this topic that I have read either include sample sizes so embarrassingly tiny it raises questions if the author even understood his statistics class (30 to 300) to simply not including numbers at all.

#2 If your audience is already pre-sold on your product when they visit your site, landing page changes won't be as meaningful. Possibly relevant to Jitbit, in this case.

#3 When you do conversion optimization, eventually you hit a number that simply becomes unbeatable. Stating the obvious, you can't convert at over 100%. Depending on traffic source/quality/intention you may find that ceiling to be lower, around 60-70%.

Re: Most AB-tests will fail

#19

Post by m4xt3r » Wed, Jun 19, 2013, 3:04 PM UTC

TallboyOne wrote: post #14

This article is full of so much wrong I can't even finish reading it. Why is this on the first page. Source: I do conversion optimization for a living.

I'm sure, everyone here would love to hear what's wrong with it. No, really, if you do that for a living, you most certanly have something to add to the discussion.

Re: Most AB-tests will fail

#20

Post by m4xt3r » Wed, Jun 19, 2013, 3:12 PM UTC

AJ007 wrote: post #18

#1 Its good to see a post using statistically significant sample sizes. Most articles on this topic that I have read either include sample sizes so embarrassingly tiny it raises questions if the author even understood his statistics class (30 to 300) to simply not including numbers at all. #2 If your audience is already pre-sold on your product when they visit your site, landing page changes won't be as meaningful. P…

#2 is totaly relevant to our Macro Recorder app.

Macro Recorder is really popular and most of the visitors are already "pre-sold". They don't care about the landing page.

We used to run tests on that page, because it has 1k daily visitors, we thought that we could test ideas more quickly. That was before we realised, that we can't rely on the results.

We even used false conclusions, based on those results, to make changes to landing pages for other products. Perhaps the biggest mistake we've made.