Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

61–70 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#61

Post by zeckalpha »

3dfan wrote: post #12

10% of the time, we choose a lever at random. The other 90% of the time, we choose the lever that has the highest expectation of rewards. There is a problem with strategies that change the distribution over time: Other factors change over time too. For example let's say over time the percentage of your traffic that comes from search engines increases. And this traffic converts better then your other traffic. And let'…

It's possible to weight conversions by frecency to account for this, rather than using frequency alone.

Re: 20 lines of code that beat A/B testing (2012)

#62

Post by zeckalpha »

mabbo wrote: post #51

This presumes a few of things about the decision being tested, many of which aren't always true. I ran a few basic A/B tests on some handscanner software used in large warehouses. The basic premise is that the user is being directed where to go and what items to collect. The customer wanted to know how changes to font size and colour of certain text would improve overall user efficiency. But the caveat was that we ha…

It's possible to assign a user to a category once a week, and use the same algorithm, rebalancing users between the categories as needed once a week.

Re: 20 lines of code that beat A/B testing (2012)

#63

Post by rco8786 »

mangeletti wrote: post #60
spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll f…

You really should question the statistical significance of those numbers.

Re: 20 lines of code that beat A/B testing (2012)

#64

Post by PaulHoule »

It is really funny how communities don't talk.

For instance, A/B testing with a 50-50 split has been baked into "business rules" framework from about as along ago as the Multi-armed bandit has been around, but nobody in that community has ever heard of the multi-armed bandit, and in the meantime, machine learning people are celebrating about the performance of NLP systems they build that are far worse than rule-based systems people were using in industry and government 15 years ago.

Re: 20 lines of code that beat A/B testing (2012)

#65

Post by jefftk »

rco8786 wrote: post #63
mangeletti wrote: post #60

Earlier quoted context omitted.

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll f…

You really should question the statistical significance of those numbers.

Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good.

So: probably better, but should have run it longer.

Re: 20 lines of code that beat A/B testing (2012)

#66

Post by [deleted] »

rco8786 wrote: post #63
mangeletti wrote: post #60

Earlier quoted context omitted.

> the difference in conversion rate is not measurable Wut? Here's the results of me changing an "add to cart" button from a branded looking maroon button to a simple yellow (Amazon style) button: http://cl.ly/0d440I3T333m That's 26% sales increase from changing a button's color. If you've got a good eye for usability, your intuition is going to lead to a lot of fun and great results with A/B testing. If not, you'll f…

You really should question the statistical significance of those numbers.

[deleted]

Re: 20 lines of code that beat A/B testing (2012)

#67

Post by cbr »

spiderfarmer wrote: post #50
butler14 wrote: post #28

Earlier quoted context omitted.

seasonality doesn't really come into it with A/B split testing - if it changes for one group it changes for both.

Let's assume your site is now working with option A. Conversion rate is 5.7%, measured over a month. You're now running an A/B for 1 week with option A and another version, option B. You get a conversion rate of 6.5% for option A and 5.9% for option B. Normally, you'd say that 6.5% is better than 5.9%. But how sure can you be if you don't control the other factors and both are performing better than before? How many…

Set your site up to be able to handle option A and option B at the same time, randomly assigning visitors to each. Now you don't need to control for anything, because you can just compare the A-visitors to the B-visitors.

In your case, option A started doing a lot better when you started running the test, which is moderately surprising, so maybe you messed something up in user assignment, tracking, or something. But if your framework is good, then it sounds like an external change. You still want to look at how A's rate and B's rate compare for the time period when you were randomizing, so 6.5% vs 5.9%, with the earlier 5.7% being more or less irrelevant.

You do need to do significance testing to see how likely you would be to get this wide a difference by chance. The easiest way to do that is to enter your #samples and #conversions into a significance calculator [1], and the p-value tells you how likely this is (assuming you didn't have any prior reason to expect one side of the experiment to be any better).

[1] Like this one https://vwo.com/ab-split-test-significance-calculator/ , but multiply the p-values it gives you by 2 to account for them incorrectly using a one-tailed test.

Re: 20 lines of code that beat A/B testing (2012)

#68

Post by mangeletti »

jefftk wrote: post #65
rco8786 wrote: post #63

Earlier quoted context omitted.

You really should question the statistical significance of those numbers.

Specifically, I get p=0.15 for those numbers. And that's assuming that they're based on running the test for a fixed amount of time instead of stopping once they look good. So: probably better, but should have run it longer.

The lady who owned the site ended up changing ecommerce platforms before the test could complete, due to issues with the software. Sadly, her "add to cart" buttons on her new site are again styled to her brand...

I wanted Optimizely to say it was 100% significant for a full week of it running before I ended the test, but the chart was interesting to me, because the conversion rate difference between the two remained the same for the entire test, rather than there being a specific period where "yellow" excelled.

Re: 20 lines of code that beat A/B testing (2012)

#69

Post by coldtea »

pdkl95 wrote: post #47
coldtea wrote: post #43

Earlier quoted context omitted.

Their trick also keeps noscript weirdos out though, which is a plus.

No, it didn't. I'm not about to run random Javascript, and the page was visible within ~3 clicks. Page that are actually broken without Javascript only indicate that the author was lazy and unprofessional. Progressive enhancement is not hard.

>Page that are actually broken without Javascript only indicate that the author was lazy and unprofessional.

Or that they have kept with the times and understand the notion of opportunity cost.

>Progressive enhancement is not hard.

No, just redundant and for marginal benefit. The train has long left that station.

Search Engines (well, those that matter) and screen readers for a11y both work with JS.

Re: 20 lines of code that beat A/B testing (2012)

#70

Post by wodenokoto »

PaulHoule wrote: post #64

It is really funny how communities don't talk. For instance, A/B testing with a 50-50 split has been baked into "business rules" framework from about as along ago as the Multi-armed bandit has been around, but nobody in that community has ever heard of the multi-armed bandit, and in the meantime, machine learning people are celebrating about the performance of NLP systems they build that are far worse than rule-based…

Which NLP system are far worse than which rule based systems?

The statement is odd for two reasons. One is that plenty of NLP is rule based, the other is that NLP isn't a form of A/B testing, which is the overall topic here..

Post reply on HN