Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B testing original design vs. random template bought from ThemeForest

candyjapan.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: A/B testing original design vs. random template bought from ThemeForest

#41

Post by iamben »

mobjack wrote: post #18
iamben wrote: post #15

Both pages would probably convert better if there was an explanation as to what happens when I put in my email address. There's also a confusing use over the word 'mailbox' - which I'd use for email, but also physical mail. The page asks for an email, immediately after saying "candy surprises in your mailbox" - if you assume no previous knowledge of what this site is going to offer, it's pretty confusing. And if it's…

From my experience, having less information on a landing page increases the email capture rate. Keep it simple with a single message all above the fold and it can improve the conversion rate by over 50%. At the same time, it doesn't make a difference either way in actual sales. At least nothing big enough that can be measured with significance. In the end, landing page optimization matters less than most people expec…

I think it depends. You can have great success with longform landers with the right product. But I agree that you should cover as much as possible as simply as possible as early as possible.

I definitely disagree that landing pages make no difference to sales and LP optimisation matters "less than most people expect". I think that wholly depends on the product (or goal) and the acquisition strategy.

Re: A/B testing original design vs. random template bought from ThemeForest

#42

Post by lunchables »

dom96 wrote: post #28
bemmu wrote: post #24

Earlier quoted context omitted.

I've always struggled to find good wording to separate physical mailboxes from electronic ones. Maybe "shipped to you twice a month" would be less vague than "in your mailbox twice a month"?

Perhaps "delivered to you twice a month" would be even better

Seems ambiguous - delivered where? My e-mail?

Re: A/B testing original design vs. random template bought from ThemeForest

#43

Post by Theodores »

fbemployee1234 wrote: post #25
Theodores wrote: post #12

I don't think this is necessarily the right conclusion. Imagine a friend with a cafe, they get the opportunity to get another cafe, same area, same clientele. With the new cafe they decide to just get some franchise, a Subway type of thing. People are familiar with it and flock there in relative droves. The original 'ma and pa' cafe is therefore deemed to be not as good as a 'template'. The other option to the franch…

How many successful businesses have you launched doing things “the right way”? The point is there are many paths to success. Maybe candy japan didn’t do things the way you think they should be done (it’s frankly rude to refer to template hacking as cargo cult programming; we’re talking design here, not programming). But despite their “primitive approach”, they’ve been successful in multiple ways. OP has taught me qui…

I get your point, but have you ever found a theme that hasn't came along with an extraordinary amount of bloat, technical debt and featuritis?

Have you ever worked for an agency where a 'theme from Theme Forest' was deemed professionally acceptable?

The fundamentals have changed for web development but nobody gets off the hamster wheel to learn the fundamentals. The web doesn't have to be an obstacle course of hacks, polyfills, libraries, frameworks and other crutches any more. It also doesn't have to be full of fancy build tools, CSS compilers and other things that make web development accessible for the first time in 20 years.

I preferred to do my own homework when I was at school, not copy what everyone else thought the answers were, changing details around.

There is an intimidating thing going on with web development and the increasing specialisation. No one can hope of being able to create a web page due to this atmosphere. But so many people - a decade or two ago - got started writing actual HTML, not assuming it is all too hard and you have to just hack someone else's work. The industry is lacking these people now and is becoming less diverse.

Frontend development isn't a creative medium if people are just using frameworks from yesteryear and taking on technical debt from themes whilst busying themselves with the latest buzzword bingo. Things like content, accessibility and document structure matter, it is no good just going for a visual design and working backwards to the inevitable 'div soup'. Something has to change.

Re: A/B testing original design vs. random template bought from ThemeForest

#44

Post by gwern »

arthurofbabylon wrote: post #39

The author lost credibility by describing statistical significance as unecessary for making informed decisions. In this context, that’s just not correct.

No, he's correct. This is a decision theory problem. Controlling false positives or false negatives at some arbitrary significance level like 0.05 is completely and utterly irrelevant. All of the debates above about which proportion test to use are irrelevant because none of them answer the question about which decision (keep current template or revert) maximizes utility (revenue). The hypothetical long-run rejection rate of a particular test in a world in which there is no difference between templates (which is a priori known to be false), has little or nothing to do with that.

Here, since he has no strong prior probability of the new version being bad, the switching costs are already sunk, and the evidence increases the posterior probability of the new version being better, and experimentation involves a high opportunity cost while being unlikely to yield a result indicating the new version is noticeably worse, keeping the new template probably is the correct decision.

Re: A/B testing original design vs. random template bought from ThemeForest

#45

Post by closed »

no_gravity wrote: post #2

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…

I don't think that's the correct use of prop.test for this question. When you give it two numbers (success, total), it tests against the null that chance of success is 50%.

Here, we want to test whether p(success|cond) differs across conditions, not whether prob(cond|success) is 50%.

This distinction is important because when p(success|cond) for some cond is low, its variance is also very small, but prop.test(39,67) doesn't reflect this. That could be 67 successes out of a small sample (and high chance of success), or out of a huge sample (and low chance of success).

Edit: whoops, I didn't notice other comments point out this issue

Re: A/B testing original design vs. random template bought from ThemeForest

#47

Post by fbemployee1234 »

Theodores wrote: post #43
fbemployee1234 wrote: post #25

Earlier quoted context omitted.

How many successful businesses have you launched doing things “the right way”? The point is there are many paths to success. Maybe candy japan didn’t do things the way you think they should be done (it’s frankly rude to refer to template hacking as cargo cult programming; we’re talking design here, not programming). But despite their “primitive approach”, they’ve been successful in multiple ways. OP has taught me qui…

I get your point, but have you ever found a theme that hasn't came along with an extraordinary amount of bloat, technical debt and featuritis? Have you ever worked for an agency where a 'theme from Theme Forest' was deemed professionally acceptable? The fundamentals have changed for web development but nobody gets off the hamster wheel to learn the fundamentals. The web doesn't have to be an obstacle course of hacks,…

You make a principled point that I respect. I’m generally trying to get shit done. They aren’t always at odds with each other, but often are.

Personally, I hate web design and layout, so I want to do as little of it as possible. But I know I need a good looking site to be taken seriously. Templates have helped me out and generally last long enough that I can afford to hire a “real” designer.

Also, templates are $50 for a beautiful one. There are 2-3 OOM difference in cost.

Post reply on HN