How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2% • Hacker News Board

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#11

Post by Datasta » Thu, Feb 03, 2011, 7:05 PM UTC

bermanoid wrote: post #8

Datasta wrote: post #5

Earlier quoted context omitted.

I cited 95% confidence, which was calculated by Visual Website Optimizer. I wonder how they calculate their confidence level.

Never used Visual Website Optimizer, but chances are, whatever they do leaves you vulnerable to the flaw mentioned in the article that swalberg linked. Basically, every time you peek at the results of a split test to see if the confidence level is high enough, you make your split test more prone to error. There are ways to adjust for this which allow you to continuously monitor the test and stop if it hits a threshol…

Actually, I had planned to test 2,000 views from the outset. It went a little over because I didn't stop it fast enough. I do agree that you are vulnerable to influencing results if you stop the test as soon as you hit your confidence level

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#12

Post by Datasta » Thu, Feb 03, 2011, 7:07 PM UTC

alain94040 wrote: post #7

Not sure if it's significant yet. What he really got was: 9 people click on the default form, 5 people on variation one, and 14 people on variation two. These are very low numbers no matter what way you look at them.

Yes, those numbers are "lowish" but remember, it was only on 2,068 uniques. We plan on running the test again soon with another 2,000 uniques to see if timing had anything to do with it.

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#13

Post by michael_dorfman » Thu, Feb 03, 2011, 7:28 PM UTC

I found it very amusing that after the long post demonstrating that the "social proof" line was cutting his sign-ups in half, there was a box asking me to sign up which included the social proof.

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#14

Post by WillyF » Thu, Feb 03, 2011, 7:31 PM UTC

It looks to me as though they should try the original headline without the social proof. Since the Control beat Variation 1, but Variation 2 beat both, that would seem to be a logical next step.

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#15

Post by Datasta » Thu, Feb 03, 2011, 7:32 PM UTC

WillyF wrote: post #14

It looks to me as though they should try the original headline without the social proof. Since the Control beat Variation 1, but Variation 2 beat both, that would seem to be a logical next step.

I agree. That's on my agenda :-D

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#16

Post by Datasta » Thu, Feb 03, 2011, 7:33 PM UTC

michael_dorfman wrote: post #13

I found it very amusing that after the long post demonstrating that the "social proof" line was cutting his sign-ups in half, there was a box asking me to sign up which included the social proof.

That's because I haven't split-tested that form yet. I'm currently preparing to run that one right now.

Re: How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

#17

Post by slimier » Thu, Feb 03, 2011, 9:14 PM UTC

WillyF wrote: post #14

It looks to me as though they should try the original headline without the social proof. Since the Control beat Variation 1, but Variation 2 beat both, that would seem to be a logical next step.

Agreed. I feel the majority of the science behind this is simply keep things clean and minimal. I would hypothesise 'Get Email Updates' would do better than 'Get Free Email Updates', too.

I don't believe a statistic offers any significant social proof. Lots of faceless people i don't know have signed up. So what?