Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What works in e-commerce – A meta-analysis of online experiments [pdf]

qubit.com

Post reply on HN
Print view View original

11–20 of 90 posts

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#11

Post by joe-stanton »

This is a really useful article. It's a shame that so much development time is wasted on large numbers of fruitless optimisations just because they are "easy" (eg. tweaking the colour of a CTA).

That being said, I'm surprised many of the results are so negative. It would be great to also see the max uplift achieved for each category. A number of retailers I've worked with have been able to beat these uplifts by quite a bit. I wonder if it might be significantly skewed by the kind of clients Qubit has?

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#12

Post by SerLava »

iagovar wrote: post #4

It's good to see an analysis, albeit most of this info was common knowledge, maybe except call to action buttons causing a decrease. I thought it was the opposite. Sometimes factoring all variables when doing testing becomes impossible.

It sounds like the experiment there wasn't to add call to action buttons, but rather to change the wording to be more action-suggestive.

And they might have flubbed them - I see people go way too far on that sometimes. Changing it from something reasonable that matched the site, to something over the top and not fitting.

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#13

Post by gruez »

tomarr wrote: post #6

If you want to see pretty much all the most effective measures in action, go to a ticket reseller like Viagogo and follow something through to the basket. It is both impressive and amazing the amount of psychological steers there are on the site.

Or flight/hotel/car rental booking websites. Google flights is a notable exception.

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#14

Post by Silhouette »

joe-stanton wrote: post #11

This is a really useful article. It's a shame that so much development time is wasted on large numbers of fruitless optimisations just because they are "easy" (eg. tweaking the colour of a CTA). That being said, I'm surprised many of the results are so negative. It would be great to also see the max uplift achieved for each category. A number of retailers I've worked with have been able to beat these uplifts by quite…

It would be great to see the max uplift achieved for each category

Indeed. What matters most with these kinds of experiments isn't really the average results, but what is possible and the distribution among beneficial results only. After all, the whole point of A/B testing is to try experiments and then either keep the changes if they improve results or stay with what you've already got if the changes didn't bring an improvement. Surely all the treatments that led to negative changes would just have been discarded in practice? It's still important to see the full picture as well, if only to guide decisions about which experiments are even worth trying, but I think there's another side that doesn't fully come through here.

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#15

Post by ifcologne »

Lot's of interesting data and some findings I didn't expected. Thanks.

What I have missed in this paper is the impact of customer reviews (a 4.5 ranking has a higher uplift potential than a 5.0 star ranking - according to some studies). And the number of reviews has impact as well. Not enough studies around for a meta analysis?

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#16

Post by brazzy »

mrweasel wrote: post #9
sweezyjeezy wrote: post #7

Earlier quoted context omitted.

This stuff is less common knowledge than you might expect. When bringing on new clients at Qubit (particularly smaller ones), we still find so many of them obsessing over tiny UI changes expecting it to make an impact. Much of the e-commerce industry has bought into the idea that the cosmetics of a site is the most important thing to get right.

I believe that the main focus of any ecommerce site should be pricing, availability and having the payment and delivery options your customers expects. E-commerce sites want to "engage" with their customers, but fail to realise that the customers don't want a relationship. They just want whatever product you're selling as quickly and cheaply as possible. Most of you customers will be coming via price comparison sites…

> For most e-commerce sites the customers aren't going to stay long enough to notice imperfections in the UI. > What you do need is a dead simple checkout (no signup required)

These two statements are contradicting each other, because "imperfections" in the UI can in fact greatly affect the perceived simplicity of the checkout process - and optimizing that is a large part of what people want to achieve with A/B testing.

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#18

Post by Silhouette »

mrweasel wrote: post #9
sweezyjeezy wrote: post #7

Earlier quoted context omitted.

This stuff is less common knowledge than you might expect. When bringing on new clients at Qubit (particularly smaller ones), we still find so many of them obsessing over tiny UI changes expecting it to make an impact. Much of the e-commerce industry has bought into the idea that the cosmetics of a site is the most important thing to get right.

I believe that the main focus of any ecommerce site should be pricing, availability and having the payment and delivery options your customers expects. E-commerce sites want to "engage" with their customers, but fail to realise that the customers don't want a relationship. They just want whatever product you're selling as quickly and cheaply as possible. Most of you customers will be coming via price comparison sites…

You seem to be taking a very narrow definition of e-commerce here. What you say seems plausible enough for sites selling low value or commodity products to casual purchasers, but I wouldn't assume the same situation for someone selling luxury products, services, digital subscriptions, etc.

Re: What works in e-commerce – A meta-analysis of online experiments [pdf]

#19

Post by jacquesm »

Silhouette wrote: post #14
joe-stanton wrote: post #11

This is a really useful article. It's a shame that so much development time is wasted on large numbers of fruitless optimisations just because they are "easy" (eg. tweaking the colour of a CTA). That being said, I'm surprised many of the results are so negative. It would be great to also see the max uplift achieved for each category. A number of retailers I've worked with have been able to beat these uplifts by quite…

It would be great to see the max uplift achieved for each category Indeed. What matters most with these kinds of experiments isn't really the average results, but what is possible and the distribution among beneficial results only . After all, the whole point of A/B testing is to try experiments and then either keep the changes if they improve results or stay with what you've already got if the changes didn't bring a…

I think the big error in A/B testing is that expectations are quite often very unrealistic. Designers typically have a reasonably good idea about what will work and what will not. Finding 'million dollar buttons' is rare. Of course a couple of percent or even 10's of percents of improvement is nothing to sneeze at. But thinking that by A/B testing forever you're going to make a shrub grow into a tree is imo not realistic. Aside from the detail that a continuously changing user interface is often in itself a barrier to sales.

Ironically, the companies that have benefited most from A/B testing were the ones that were doing a terrible job of it in the first place so then there is lots of low hanging fruit making the consultants look good.

Yet another item often missed: A/B testing success is a direct function of the length of the lever you are pulling. If that lever commands billions of dollars then it is easy to make it pay for itself. But if you're trying to turn $10000 into $11500 then you likely are wasting your time.

Post reply on HN