Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most AB-tests will fail

jitbit.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Most AB-tests will fail

#31

Post by pbiggar »

I think this post and most posts on the topic are missing a major advantage of A/B testing. Sure, one of its uses is to experiment and try to keep optimizing everything. The other use of A/B testing is to prevent regressions in a statistically rigorous way.

Basically, if you can afford it, every change to the website could be A/B tested, and you'd know that the change doesn't negatively impact the conversion rate. Sometime's its really useful to know that you can safely change the button from red to a colour that matches your palette better, and "no change" is a great result.

Re: Most AB-tests will fail

#32

Post by JacobJans »

pbiggar wrote: post #31

I think this post and most posts on the topic are missing a major advantage of A/B testing. Sure, one of its uses is to experiment and try to keep optimizing everything. The other use of A/B testing is to prevent regressions in a statistically rigorous way. Basically, if you can afford it, every change to the website could be A/B tested, and you'd know that the change doesn't negatively impact the conversion rate. So…

I've been using this thinking recently to great advantage.

My basic strategy has been to make the design more beautiful, and if the conversion rate stays the same or improves, I've succeeded.

This way, I've been able to significantly improve the aesthetic appeal of many of my sites without risking the bottom line.

Re: Most AB-tests will fail

#33

Post by mikebell »

Just because the A/B test doesn't produce the results you expected doesn't mean it produced no results. Research scientists who don't prove their hypothesis don't say they learned nothing from the experiments, they just write a different paper.

Just to pick one example from this post; the whole-site redesign didn't change conversions. Did it change something else? More importantly, you now know that investing effort in a full re-design didn't really have any effect. Next time someone suggests changing the site, you have evidence that says it may not be worth the engineering cost, and maybe its better to focus somewhere else first.

Re: Most AB-tests will fail

#34

Post by aresant »

m4xt3r wrote: post #29
aresant wrote: post #26

Most A/B testing will fail when you're testing the wrong elements. I have a challenge for you. The examples you list are "top" of the funnel tests: buttons for engagement, a homepage redesign. But when I go deeper into your funnel like your pricing grid page and cart checkout page, I see lots of tweaks that, in my experience, can drive the significant gains that you are looking for. I'm going to use this page as my s…

Thanls a lot for taking your time to write this awesome comment! The thing is we have very little control over the checkout area, since it is hosted by our payment provider and we cannot run tests there. We are not happy about it and we will move to Stripe as soon as it will be available outside of the US. Really, thnaks a lot for this. I will look into this right now, maybe we could fix some of those. EDIT: We've re…

My pleasure and hope that it helps!

If you guys are doing decent scale on sales it's 100% worth building your own checkout process so you can really drill down your high-intent traffic, or at least testing stripe / etc (or whatever other solutions are available in your market)

Re: Most AB-tests will fail

#35

Post by LanceJones »

Interesting headline on the HN post. Joanna and I at Copy Hackers respectfully disagree. A properly-designed split test will ALWAYS produce a result -- you'll generate negative, positive, or neutral lift. And no matter what type of lift you achieve, you can ALWAYS learn from the test.

We just concluded an 11-site split test where we changed nothing but the home page headline -- written as a value proposition -- and we produced 9 of 11 "winning" tests, with an average conversion increase of 34% on the primary KPI (and the 34% includes the "losing" tests).

Bottom line is (from our many years of testing copy)... you have to change meaningful things to create conversion lift. And "meaningful" means in the eyes of your visitors.

Re: Most AB-tests will fail

#36

Post by drhayes9 »

> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it.

I'm sensitive to hyperbole, and I know he later refines his statement, but this is not true. Google Ads change what seem to be insignificant details by tiny amounts all the time (luminance on links, vertical space below the search bar, padding around sitelinks) and derive significant revenue changes.

I think one issue is that most sites' A/B tests results are difficult to distinguish from noise while Google has what you might call a significant user base.

Re: Most AB-tests will fail

#37

Post by jordo37 »

m4xt3r wrote: post #29
aresant wrote: post #26

Most A/B testing will fail when you're testing the wrong elements. I have a challenge for you. The examples you list are "top" of the funnel tests: buttons for engagement, a homepage redesign. But when I go deeper into your funnel like your pricing grid page and cart checkout page, I see lots of tweaks that, in my experience, can drive the significant gains that you are looking for. I'm going to use this page as my s…

Thanls a lot for taking your time to write this awesome comment! The thing is we have very little control over the checkout area, since it is hosted by our payment provider and we cannot run tests there. We are not happy about it and we will move to Stripe as soon as it will be available outside of the US. Really, thnaks a lot for this. I will look into this right now, maybe we could fix some of those. EDIT: We've re…

I would also highly encourage you to look at Balanced or Stripe for checkout - the integration isn't too bad, the pricing is great and you have full control over the look and feel of your checkout pages.

Also, great tip on the CVV2 info, realized we didn't have it in our system either.

Re: Most AB-tests will fail

#38

Post by janesvilleseo »

drhayes9 wrote: post #36

> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it. I'm sensitive to hyperbole, and I know he later refines his statement, but this is not true. Google Ads change what seem to be insignificant details by tiny amounts all the time (luminance on links, vertical space below the search bar, padding around sitelinks) and derive significant revenue…

I agree with you above statement. However, in regards to your example. I often wonder how much of the increase is due to Google overcoming ad blindness and how much is because of the ads are truly more engaging.

Re: Most AB-tests will fail

#39

Post by drhayes9 »

janesvilleseo wrote: post #38
drhayes9 wrote: post #36

> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it. I'm sensitive to hyperbole, and I know he later refines his statement, but this is not true. Google Ads change what seem to be insignificant details by tiny amounts all the time (luminance on links, vertical space below the search bar, padding around sitelinks) and derive significant revenue…

I agree with you above statement. However, in regards to your example. I often wonder how much of the increase is due to Google overcoming ad blindness and how much is because of the ads are truly more engaging.

Good question. That's where longer-term measurement comes in. Overcoming ad blindness is a transitory thing that goes away after a period of time while engagement tends to stick (as measured by CTR, say).

Re: Most AB-tests will fail

#40

Post by MichaelGG »

m4xt3r wrote: post #29
aresant wrote: post #26

Most A/B testing will fail when you're testing the wrong elements. I have a challenge for you. The examples you list are "top" of the funnel tests: buttons for engagement, a homepage redesign. But when I go deeper into your funnel like your pricing grid page and cart checkout page, I see lots of tweaks that, in my experience, can drive the significant gains that you are looking for. I'm going to use this page as my s…

Thanls a lot for taking your time to write this awesome comment! The thing is we have very little control over the checkout area, since it is hosted by our payment provider and we cannot run tests there. We are not happy about it and we will move to Stripe as soon as it will be available outside of the US. Really, thnaks a lot for this. I will look into this right now, maybe we could fix some of those. EDIT: We've re…

Braintree is available outside the US, and they offer the client-side JS to you can control the experience and dodge PCI requirements.
Post reply on HN