Most AB-tests will fail • Hacker News Board

Re: Most AB-tests will fail

#41

Post by hackiavelli » Wed, Jun 19, 2013, 9:26 PM UTC

aresant wrote: post #26

Most A/B testing will fail when you're testing the wrong elements. I have a challenge for you. The examples you list are "top" of the funnel tests: buttons for engagement, a homepage redesign. But when I go deeper into your funnel like your pricing grid page and cart checkout page, I see lots of tweaks that, in my experience, can drive the significant gains that you are looking for. I'm going to use this page as my s…

I'm always resistant to putting up "trust logos" on a website. It's like training users to fall for phishing scams: you can trust me because I say you can trust me. Sure, the user can click the link to the verifying site, but how many do? Of those, how many would be able to spot it if you were redirecting them to a look-alike you set up?

I'd much rather train users to trust their SSL connection based on what their browser says rather than what I do.

Re: Most AB-tests will fail

#42

Post by alohahacker » Thu, Jun 20, 2013, 1:10 AM UTC

"No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it."

I remember Josh the founder of Omniture said how they first got hired by ebay back in the day. Changing the color of the ebay homepage and button to yellow helped ebay increase sales by 15%. That slight increase on a heavily trafficked website resulted in millions of dollars in increased sales.

Re: Most AB-tests will fail

#43

Post by justinmarsan » Thu, Jun 20, 2013, 10:14 AM UTC

m4xt3r wrote: post #10

Slix wrote: post #2

> No company have ever increased the revenue by changing an insignificant detail like a button color. None. Zero. Get over it. This contradicts the article about A/B testing by Wired, which used the Obama campaign as an example. http://www.wired.com/business/2012/04/ff_abtesting

Alright, that was a bold statement on my side. That's true for 99.9% of the companies, not 100%. Again, most of us don't have enough traffic to make successful buton color tests.

Have you tried MAB ? It seems like the best way to do A/B testing on small-traffic pages, let it run forever without losing much due to the reduced frequency of poorly performing variantes.

Setup your test with 3 or 4 colors, the programm will automatically pick the best one over time, so it doesn't matter how long you let it run, be it weeks months or years, and when you don't want to test it anymore you pick the one that performed the best out of the many visitors you've had over a large period.

http://en.wikipedia.org/wiki/Multi-armed_bandit