Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

51–60 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#51

Post by mabbo »

This presumes a few of things about the decision being tested, many of which aren't always true.

I ran a few basic A/B tests on some handscanner software used in large warehouses. The basic premise is that the user is being directed where to go and what items to collect. The customer wanted to know how changes to font size and colour of certain text would improve overall user efficiency. But the caveat was that we had to present a consistent experience to the user- you can't change the font size every 10 seconds or it will definitely degrade user experience!

My point is that it sounds as though the 1-armed bandit will probably work great provided the test is short, simple, and the choice can be re-made often without impacting users.

Re: 20 lines of code that beat A/B testing (2012)

#52

Post by spiderfarmer »

darkxanthos wrote: post #46
spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

My professional advice to people who make these unmeasurable changes is: Then don't do them. The worst part about unmeasurable changes is you can't verify if it has a negative effect either and you're essentially saying "I'm focusing on making this change when it will have negligible impact on the business." You have better things to do.

I agree, every article about A/B testing is talking about making small changes and measure them, while you should be measuring only big changes or nothing at all.

Re: 20 lines of code that beat A/B testing (2012)

#53

Post by spiderfarmer »

teekert wrote: post #25
spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc." I'd serve them at the same time, randomize which clients see which one.

Ofcourse, that's what A/B testing is about. But this only works if you have lots of traffic. The smaller the amount of traffic, the bigger the influence of offsite factors. Also, the results become more difficult to explain: https://news.ycombinator.com/item?id=11438098

Re: 20 lines of code that beat A/B testing (2012)

#54

Post by TeMPOraL »

coldtea wrote: post #43
pdkl95 wrote: post #9

Earlier quoted context omitted.

Apparently some sort of useless fade-in effect is more important than the page content. This is the fix: #main { visibility: visible !important; }

Their trick also keeps noscript weirdos out though, which is a plus.

Actually, weirdos are people making such pages. There is no valid reason for the content here to be not visible without JavaScript on. It's laziness and/or stupidity.

Re: 20 lines of code that beat A/B testing (2012)

#56

Post by snowwrestler »

TuringTest wrote: post #32
kevin_nisbet wrote: post #15

Earlier quoted context omitted.

I don't know if this is the case if I understand this algorithm correctly. Say Yellow is 50%, and Green is a 65% success rate after the behavior change, but green is 30% before the behavior change. By sending 90% of traffic towards yellow, it's ratio will normalize towards the 50% once it has enough traffic. By sending 10% of traffic randomly, eventually the green option will reach 51%, and start taking a majority of…

> Also, to me, the concept of A/B testing certain things may also have an undesired consequence. For example, I order from amazon every day, but today the but button is blue, what does that actually mean? And I go back to the site later and it's yellow again. There are still many people who get confused by seemingly innocuous changes with the way their computer interacts with them. Proper A/B tests are supposed to be…

Until that particular test ends, and another one begins.

Re: 20 lines of code that beat A/B testing (2012)

#60

Post by mangeletti »

spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

> the difference in conversion rate is not measurable

Wut?

Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m

That's 26% sales increase from changing a button's color.

If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll futz with things that make no difference most of the time and eventually give up. Experimentation is not about testing random permutations (unless you've got an infinite amount of time). It's about coming up with a reasonable hypothesis and then testing it. I study a web page until I come to a conclusion in my mind that something could really use some improvement. Then, even though I'm sure of it in my mind, I test it, because "sure of it in my mind" is wrong about half the time.

One note to those who use Optimizely: you do need to wait at least twice as long as Optimizely thinks, because statistical significance can bounce around with too little traffic. Optimizely thought the above test was done at about 2000 views, which was far too little results to be conclusive, with only ~20 sales.

Post reply on HN