Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing original design vs. random template bought from ThemeForest

candyjapan.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: A/B testing original design vs. random template bought from ThemeForest

#11

Post by no_gravity »

muraiki wrote: post #7
no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particu…

    What you've tested using that invocation is the
    null hypothesis that the underlying probability
    of 39/67 is 0.5.
Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time".

    If you want to perform a test of a difference of two proportions, you need to do:
    prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))
Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22.

As long as the group sizes are equal, the test is not very sensitive to the sizes.

Re: A/B testing original design vs. random template bought from ThemeForest

#12

Post by Theodores »

I don't think this is necessarily the right conclusion.

Imagine a friend with a cafe, they get the opportunity to get another cafe, same area, same clientele. With the new cafe they decide to just get some franchise, a Subway type of thing. People are familiar with it and flock there in relative droves. The original 'ma and pa' cafe is therefore deemed to be not as good as a 'template'.

The other option to the franchise could have been to have gone with the 'ma and pa' offering, same deal.

But that is not all of the choices. You could actually create something new, not hold with what you have or chuck it in and go with the 'franchise'.

With website design it is very easy for people to chuck it in and go with a Themeforest effort, Shopify, Squarespace and the like.

But a lot of 'new ingredients' have came along with CSS Grid, semantic markup and much else that the themes, 'serviced websites' and the like are not up to speed with yet and show no signs of wanting to implement.

It is also possible to build out more than just a landing page from scratch in two weeks with the new tools. This involves learning instead of botching someone else's floats, divs and margin hacks. We have Cargo Cult programming and in my opinion this story is just another example of this.

I would recommend anyone else in this position to start content first, not buying a theme and then taking the photos needed to push in the theme because the boxes are there already. This is a backwards, 'design led' process and there are plenty of good reasons to go content first, then structure it properly, spend a day or two with CSS grid, then put the existing branding on there. A static HTML page is a good start, a working prototype instead of a PDF mockup.

Then, instead of the A/B testing, ask an honest person who knows a thing or two and won't bullshit you. They will ask 'why have you done that' questions which will get the content in shape. Thereafter, once you have a go at doing it you can spot what you like elsewhere and learn from it rather than cargo-cult-copy-paste it to never be confident of anything.

Re: A/B testing original design vs. random template bought from ThemeForest

#14

Post by no_gravity »

bemmu wrote: post #9
artiscode wrote: post #8

Earlier quoted context omitted.

Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.

Yes, I don't have much data to work with, and was also surprised that the calculator considered this significant. But even without significance, I assume it still makes sense to go with the winner?

Does the calculator really use the word "significant"? I don't see it. I am not sure how to interpret the language it uses.

As for going with the winner: Yes, if the test result (39/28) is the only information you have and there are only two choices (go with winner / go with loser) then it makes sense to go with the winner.

Re: A/B testing original design vs. random template bought from ThemeForest

#15

Post by iamben »

Both pages would probably convert better if there was an explanation as to what happens when I put in my email address. There's also a confusing use over the word 'mailbox' - which I'd use for email, but also physical mail. The page asks for an email, immediately after saying "candy surprises in your mailbox" - if you assume no previous knowledge of what this site is going to offer, it's pretty confusing. And if it's confusing, you're less likely to get people to sign up.

Without having a proper look at and understanding the site, I'm not sure whether that email is a leadgen form, or start of the sign up process, but either way you could probably increase conversion with some better explanatory copy.

Interesting read though! Thanks for sharing!

Re: A/B testing original design vs. random template bought from ThemeForest

#16

Post by nls »

It's definitely not just the design itself. The CTA (your email submit) of the new landing page is better positioned throughout. You have also added urgency ("time until next shipment", "subscribe now before time runs out"). You can also see up to 3 previous candy boxes now, whereas before, you'd have to click through. All of those contribute to the conversion rate of your new landing page.

Your old landing page actually contains some great elements that are missing from the new one, like the reviews, the explanation anime video and the "new tastes only available in Japan", which seems a great feature to me. I'd definitely try and add those back in, and split test that new version against the current winner.

Re: A/B testing original design vs. random template bought from ThemeForest

#17

Post by ajharrison »

Willing to bet that with some improvements to the original design the results would've been even better.

For example:

– Your navigation at the top is illegible. Theres very little contrast and it's hard to read. Make that more clear, higher contrast.

- Your main CTA ("Japanese candy surprises in your mailbox twice a month." [Email]) is also hard to read because of the orange and the custom font. Swap out the orange for a more vibrant color (maybe the orange color of the button) and use a nicer font that's easier to read.

- MAKE THE FONT SIZE BIGGER

- Add in some of the elements you added in the new design: When the next box is going out, etc.)

Your original design had an air of authenticity and honesty to it. The bought template seems like a copycat.

Keep (and improve!) the original!

Re: A/B testing original design vs. random template bought from ThemeForest

#18

Post by mobjack »

iamben wrote: post #15

Both pages would probably convert better if there was an explanation as to what happens when I put in my email address. There's also a confusing use over the word 'mailbox' - which I'd use for email, but also physical mail. The page asks for an email, immediately after saying "candy surprises in your mailbox" - if you assume no previous knowledge of what this site is going to offer, it's pretty confusing. And if it's…

From my experience, having less information on a landing page increases the email capture rate. Keep it simple with a single message all above the fold and it can improve the conversion rate by over 50%.

At the same time, it doesn't make a difference either way in actual sales. At least nothing big enough that can be measured with significance.

In the end, landing page optimization matters less than most people expect. It is better to work on other parts of the website.

Re: A/B testing original design vs. random template bought from ThemeForest

#19

Post by rwilson4 »

no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

Using the techniques described on my blog [0], which are ideal for KPIs like conversion rates and small sample sizes (since no Gaussian approximation is made), I get a p-value of 0.177, which is not significant. The observed treatment effect is a 36.8% lift in conversion rate, but a confidence interval on this effect has endpoints -10.7% and +97.4%. Anything in that range would be considered consistent with the observed result at a 0.05 significance threshold.

With 6000 total impressions and a 50/50 split, the experiment is only able reliably to detect a 74% lift in conversion rate (with power = 80%).

If you want to rigorously determine the impact, decide what effect size you hope to see. Use a power calculator to decide the sample size needed to detect that effect size. Administer the test, waiting to acquire the planned sample size. When analyzing, be sure to compute a p-value and a confidence interval on the treatment effect.

[0] https://www.adventuresinwhy.com/post/ab-testing-random-sampl...

Re: A/B testing original design vs. random template bought from ThemeForest

#20

Post by muraiki »

no_gravity wrote: post #11
muraiki wrote: post #7

Earlier quoted context omitted.

That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particu…

What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time". If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_…

I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same.

Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!

Post reply on HN