How many users do you need to start split testing?

#1

Post by lionesscg » Thu, Apr 24, 2014, 4:56 PM UTC

We recently launched this website, wannity.com, and are having pretty much visits than we expected (around 1k/month). We have low conversion rates right now, and we're wondering if it's time to start split testing, or the number of users is not enough to start making decisions after these tests?

Thanks.

Re: How many users do you need to start split testing?

#2

Post by esquivalience » Thu, Apr 24, 2014, 4:59 PM UTC

Surely you can split-test at any level, it just takes longer for useful data to build up?

Then it just becomes a question of confidence interval.

Re: How many users do you need to start split testing?

#3

Post by ondiekijunior » Thu, Apr 24, 2014, 5:14 PM UTC

I would say you have more than sufficient numbers to do all the testing.

Re: How many users do you need to start split testing?

#4

Post by karolisd » Thu, Apr 24, 2014, 6:44 PM UTC

If you do many small tests, you can pick confidence level less that 95%. Even if you call tests at 70% confidence, as long as you are iterating rapidly, you'll end up ahead in the long run. Also, you can revisit the same tests later. As long as your testing ideas aren't prohibitively expensive in time and implementation, there's very little risk at calling tests below 95% confidence.

Re: How many users do you need to start split testing?

#5

Post by gxs54 » Thu, Apr 24, 2014, 7:59 PM UTC

esquivalience wrote: post #2

Surely you can split-test at any level, it just takes longer for useful data to build up? Then it just becomes a question of confidence interval.

I agree, you could split test with a lot less users.

Re: How many users do you need to start split testing?

#6

Post by lionesscg » Fri, Apr 25, 2014, 2:24 PM UTC

Ok, thanks so much for the quick replies everybody. Really appreciate it.

Re: How many users do you need to start split testing?

#7

Post by siddharthdeswal » Fri, Apr 25, 2014, 9:44 PM UTC

Right now, instead of split testing, do usability testing. Get 5 to 10 people who are close to the kind of people you expect to be visiting your website (or buying from you), and ask them open-ended questions.

Believe me, at the stage you're in, you'll learn far more than whatever split-testing you can do right now.

Here are a few questions I could think of (in no order):

1) On a previous visit to the website, you saw a bookshelf that was in the form of a bent arrow. What would you do if you wanted to buy it now?

2) If you wanted to keep up with all the new products on Wannity, can you find a way to do that?

3) Why would you not share a product you like in Wannity on your social networks?

4) What could be made better about the Checkout process?

5) Why would you not buy from Wannity?

Also, one small issue: I can't see the complete form in 1366 x 768 (http://puu.sh/8nkKK.png). If most of your buyers/visitors are on larger heights, you're good. Else maybe a bit of restructuring is in order.

Re: How many users do you need to start split testing?

#8

Post by lionesscg » Mon, Apr 28, 2014, 11:32 AM UTC

siddharthdeswal wrote: post #7

Right now, instead of split testing, do usability testing. Get 5 to 10 people who are close to the kind of people you expect to be visiting your website (or buying from you), and ask them open-ended questions. Believe me, at the stage you're in, you'll learn far more than whatever split-testing you can do right now. Here are a few questions I could think of (in no order): 1) On a previous visit to the website, you sa…

Thank you so much siddharthdeswal. We made a Google form in order to get qualitative data, but this will be much more productive. Thanks for the bug reporting also.