Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing original design vs. random template bought from ThemeForest

candyjapan.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: A/B testing original design vs. random template bought from ThemeForest

#31

Post by eutectic »

no_gravity wrote: post #10
bemmu wrote: post #6

Earlier quoted context omitted.

I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.

The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate". I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.

You get 91% if you put a uniform prior on the proportion coming from each alternative.

Re: A/B testing original design vs. random template bought from ThemeForest

#32

Post by funkaster »

bemmu wrote: post #6
no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.

If you had 3000 visits, shouldn't that be 1500 -> 28 and 1500 -> 39? (assuming you're doing a uniform split of both groups)

Re: A/B testing original design vs. random template bought from ThemeForest

#33

Post by cle »

codetrotter wrote: post #29
no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

They said the following in the beginning though: > For example if you want to test a tweak that results in 5% more conversions, you need about 3000 sales to detect it! For Candy Japan this would mean waiting for about 10 years for the test to complete. But they want to do something still to try and improve sales. Seems reasonable even if not scientific.

This is not the right approach to take then. There are lots of other approaches to decision making outside of hypothesis testing, use them! This is not an appropriate use of hypothesis testing and can very much lead you toward making the WRONG decision.

For example, with such small numbers, there isn't much value in aggregate statistics. It would take a day or two to go through each one individually and see what happened, and you'd probably learn way more about your customers.

Re: A/B testing original design vs. random template bought from ThemeForest

#34

Post by edoceo »

dom96 wrote: post #28
bemmu wrote: post #24

Earlier quoted context omitted.

I've always struggled to find good wording to separate physical mailboxes from electronic ones. Maybe "shipped to you twice a month" would be less vague than "in your mailbox twice a month"?

Perhaps "delivered to you twice a month" would be even better

Delivered to your door

Re: A/B testing original design vs. random template bought from ThemeForest

#35

Post by popinman322 »

no_gravity wrote: post #10
bemmu wrote: post #6

Earlier quoted context omitted.

I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.

The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate". I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.

It looks like the difference of two beta distributions based on the visualization.

So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2?

I'm open to correction here.

Re: A/B testing original design vs. random template bought from ThemeForest

#36

Post by kqr »

olavolav wrote: post #5
no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

Whatever test you do, you would need to know the total number of visitors in each group, right? And unless I missed it the article doesn't state those numbers. Intuitively, the numbers you quoted would be more significant the bigger the test and control groups are.

That's not very intuitive for me. Let's do some limit analysis: imagine the groups were one million sessions each, but the convertions in the groups were only one and two people respectively. Wouldn't this result seem like the result of random chance?

The conversion rate is basically one in a million in both cases.

Re: A/B testing original design vs. random template bought from ThemeForest

#37

Post by chiefalchemist »

Small sample size aside, they used the wrong KPI (imho). This needs to be tested over time. That is, which one retains better? Focusing on conversion's is a fool's errand. That is, you convert better but then if churn is higher you could end up at net loss. That's not ideal, obviously.

I think also, you'd have to look at referrals (if you have them). Perhaps a lower subscription rate actually led to more sales because (for some reason) those subscribers like to tell their friends.

The analysis here is too shallow.

Re: A/B testing original design vs. random template bought from ThemeForest

#38

Post by rchaud »

bemmu wrote: post #6
no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.

If you have Google Analyics on your site, you can use the Unique Pageviews metric for each of the two page variations and use that as the metric, instead of arbitrarily assigning 3000 views to both.

Re: A/B testing original design vs. random template bought from ThemeForest

#40

Post by iamben »

bemmu wrote: post #24
iamben wrote: post #15

Both pages would probably convert better if there was an explanation as to what happens when I put in my email address. There's also a confusing use over the word 'mailbox' - which I'd use for email, but also physical mail. The page asks for an email, immediately after saying "candy surprises in your mailbox" - if you assume no previous knowledge of what this site is going to offer, it's pretty confusing. And if it's…

I've always struggled to find good wording to separate physical mailboxes from electronic ones. Maybe "shipped to you twice a month" would be less vague than "in your mailbox twice a month"?

Yep, or you could describe the physical product and make sure it's clear what people are putting their email address in for.

Like "we ship a box of surprising Japanese candy to your house every month" then explain why you need the email.

If it was my lander I'd experiment with showing prices before you ask for an email. Just from my experience people are reticent to put in an email without knowing what the reason is.

Post reply on HN