Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

31–40 of 164 posts

Re: 20 lines of code that beat A/B testing (2012)

#31

Post by chii »

teekert wrote: post #25
spiderfarmer wrote: post #17

I did a lot of A/B testing, but I think the examples that are used in a lot of articles about A/B testing are weird. For small changes like change the color / appearance of a button, the difference in conversion rate is not measurable. Maybe if you can test with traffic in the range of >100K unique visitors (from the same sources), you can say with confidence which button performed better. But how many websites / app…

" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc." I'd serve them at the same time, randomize which clients see which one.

And it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...

Re: 20 lines of code that beat A/B testing (2012)

#32

Post by TuringTest »

kevin_nisbet wrote: post #15
3dfan wrote: post #12

10% of the time, we choose a lever at random. The other 90% of the time, we choose the lever that has the highest expectation of rewards. There is a problem with strategies that change the distribution over time: Other factors change over time too. For example let's say over time the percentage of your traffic that comes from search engines increases. And this traffic converts better then your other traffic. And let'…

I don't know if this is the case if I understand this algorithm correctly. Say Yellow is 50%, and Green is a 65% success rate after the behavior change, but green is 30% before the behavior change. By sending 90% of traffic towards yellow, it's ratio will normalize towards the 50% once it has enough traffic. By sending 10% of traffic randomly, eventually the green option will reach 51%, and start taking a majority of…

> Also, to me, the concept of A/B testing certain things may also have an undesired consequence. For example, I order from amazon every day, but today the but button is blue, what does that actually mean? And I go back to the site later and it's yellow again. There are still many people who get confused by seemingly innocuous changes with the way their computer interacts with them.

Proper A/B tests are supposed to be done on a per-unique-user basis. If you access the shop from the same device or user account, a well-done A/B test should consistently show you the same interface.

Re: 20 lines of code that beat A/B testing (2012)

#33

Post by raverbashing »

tzs wrote: post #29
raverbashing wrote: post #14

Earlier quoted context omitted.

The points raised are valid, if they matter is a different beast Even in the tests shown, conversion rate was higher for the MABA algorithms than simple A/B testing. "Oh but you get higher statistical significance!" thanks, but that doesn't pay my bills, conversion pays.

Careful. It wasn't always higher for MAB even though the tables shown there make it appear so at first. Those tables are showing the conversion rate during the test, up to the time when statistical significance is achieved. You generally then stop the test and go with the winning option for all your traffic. In the two-way test where the two paths have real conversion rates of 10% and 20%, all of the MAB variations d…

Interesting

This suggest to me that, similarly to a lot of algorithms you might want to change your parameters during training

So start with MAB-100 (RAND) and then decrease that % over time

Re: 20 lines of code that beat A/B testing (2012)

#34

Post by thenomad »

paraschopra wrote: post #22
aidanf wrote: post #13

Earlier quoted context omitted.

Here are 2 good articles that follow up on the arguments presented by VWO in that article. From the first link below: "They do make a compelling case that A/B testing is superior to one particular not very good bandit algorithm, because that particular algorithm does not take into account statistical significance. However, there are bandit algorithms that account for statistical significance." * https://www.chrisstuc…

Chris is now VWO's director of data science. We recently overhauled our stats. Here's a quick summary for that: https://vwo.com/blog/smartstats-testing-for-truth/

Interesting. Lack of Bayesian testing is one of the reasons I've never considered using VWO - nice to see that's now your main methodology.

Re: 20 lines of code that beat A/B testing (2012)

#35

Post by kriro »

raverbashing wrote: post #14
ReadingInBed wrote: post #7

I thought this was a pretty good follow up to show the strengths and weaknesses of this approach: https://vwo.com/blog/multi-armed-bandit-algorithm/ . Personally I think this approach makes a lot more sense than a/b testing especially when often people hand off the methodology to a 3rd party without knowing exactly how they work.

The points raised are valid, if they matter is a different beast Even in the tests shown, conversion rate was higher for the MABA algorithms than simple A/B testing. "Oh but you get higher statistical significance!" thanks, but that doesn't pay my bills, conversion pays.

The counterargument would be: in the long run, random positive changes in clicks don't pay the bills. Systemic changes do.

Re: 20 lines of code that beat A/B testing (2012)

#36

Post by ryporter »

This is a good overview of the multi-arm bandit problem [1], but the author is far too dismissive of A/B Testing.

First of all, the suggested approach isn't always practical. Imagine that you are testing an overhaul of your website. Do you want daily individual visitors to keep flipping back and forth as the probabilities change? I'm not sure if the author is really suggesting his approach would be a better way to run drug trials, but that's clearly ridiculous. You have to recruit a set of people to participate in the study, and then you obviously can't change what drug you're giving them during the course of experiment!

Second, it ignores the time value of completing an experiment earlier. In the exploration/exploitation tradeoff, sometimes short-term exploitation isn't nearly as valuable as wrapping up an experiment so that your team can move to new experiments (e.g., shuting down the old website in the previous example). If a company expects to have a long lifetime, then, over the a time frame measured in weeks, exploration will likely be relatively far more valuable.

[1] https://en.wikipedia.org/wiki/Multi-armed_bandit

Re: 20 lines of code that beat A/B testing (2012)

#37

Post by raverbashing »

kriro wrote: post #35
raverbashing wrote: post #14

Earlier quoted context omitted.

The points raised are valid, if they matter is a different beast Even in the tests shown, conversion rate was higher for the MABA algorithms than simple A/B testing. "Oh but you get higher statistical significance!" thanks, but that doesn't pay my bills, conversion pays.

The counterargument would be: in the long run, random positive changes in clicks don't pay the bills. Systemic changes do.

Yes and you don't get systemic change with A/B testing

I find it funny when some people think A/B (or MAB) testing will solve major usability problems

You can definitely try Genetic Programming your website to conversion, it's probably going to be fun to watch

Re: 20 lines of code that beat A/B testing (2012)

#38

Post by eru »

chii wrote: post #31
teekert wrote: post #25

Earlier quoted context omitted.

" If you have a long running test, just to gather enough traffic, changes are some other factors have changed as well, like the weather, weekdays / weekends, time of month, etc." I'd serve them at the same time, randomize which clients see which one.

And it turns out you randomly assigned B to a poorer user more often than not, and this make B worse...

That's why you do it properly randomized. Statistics tell you how much of a risk of your scenario (and similar) you are running at most.

Re: 20 lines of code that beat A/B testing (2012)

#39

Post by wtbob »

dspillett wrote: post #23
pmontra wrote: post #10

Earlier quoted context omitted.

Very funny. They have a div #main with visibility:hidden. Removing that rule from dev tools displays the full page without enabling JS. The comments block is a solved problem with CSS. Basically any grid layout toolkit does that as their very first demo.

Yeah, if you are going to do that just for a fade-in transition at least set it to hidden with an inline script block then someone with script turned off they get the content. You block rendering until everything is downloaded, but then so does your needless transition. If you are worried about it breaking in some browsers because you are modifying a tag that has not yet had its content completed so isn't accessible…

> Or, of course, just don't...

I think that's really the takeaway here. Bytes and bytes of JavaScript, slower rendering time, and for what? Just say no!

Re: 20 lines of code that beat A/B testing (2012)

#40

Post by davkap92 »

ryporter wrote: post #36

This is a good overview of the multi-arm bandit problem [1], but the author is far too dismissive of A/B Testing. First of all, the suggested approach isn't always practical. Imagine that you are testing an overhaul of your website. Do you want daily individual visitors to keep flipping back and forth as the probabilities change? I'm not sure if the author is really suggesting his approach would be a better way to ru…

Regarding your first point, not sure if author covered it, but in Google Content Experiments with multi bandit approach cookies are stored, so user who sees variation b will keep seeing b while the experiment is running
Post reply on HN