Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20 lines of code that beat A/B testing every time

stevehanov.ca

Post reply on HN
Print view View original

101–110 of 152 posts

Re: 20 lines of code that beat A/B testing every time

#101

Post by andrewreds »

btilly wrote: post #53

This is thought-provoking, which is good. However there are significant issues with the approach. 1. Real world performance varies over time. For instance there are typically daily, weekly and monthly conversion rate fluctuations. Not an issue for A/B testing, but a big issue for this approach if a random switch in direction happens at the same time that conversion fluctuations happen to head in a good direction. 2.…

re #1,

could this be solved with an exponential decay?

aka, saying that a click from 1 month ago is less valuable than someone clicking today.

By tweaking the decay you could change how quickly the algorithm will sway when the conversion rate changes.

EDIT: I just saw that rauljara suggested this below: https://news.ycombinator.com/item?id=4040230

Re: 20 lines of code that beat A/B testing every time

#102

Post by evropamx »

wisty wrote: post #23

A stevehanov.ca link? Wow, HN is getting classy again. Please more articles with code, equations and / well visualizations, and less upvoting of badly thought out infograpics (i.e. pretty numbers which would lose nothing by just being presented in a table) and far less self-help pseudo business articles please. +1 on an article does not mean "I agree". It means "I learnt something".

And I really did, so +1 from me.

Re: 20 lines of code that beat A/B testing every time

#103

Post by zader »

btilly wrote: post #90
pbreit wrote: post #84

Earlier quoted context omitted.

I don't think any of those points are very true.

I've been involved with A/B testing for nearly a decade. I assure you that none of these points are in the slightest bit hypothetical. 1. Every kind of lead gen that I have been involved with and thought to measure has large periodic fluctuations in user behavior. Measure it, people behave differently on Friday night and Monday morning. 2. If you're regularly running multiple tests at once, this should be a potential…

What do you think of Myna, in these respects? Does it suffer from the same disadvantages as other bandit optimization approaches?

http://mynaweb.com/docs/

Re: 20 lines of code that beat A/B testing every time

#104

Post by tomp »

noelwelsh wrote: post #94

First up, the sales pitch: we provide bandit optimisation SaaS at Myna: http://mynaweb.com Now, that's out of the way, let's discuss the article. I like the epsilon-greedy algorithm because it's simple to understand and implement, and easy to extend. However, to claim "The strategy that has been shown to win out time after time in practical problems is the epsilon-greedy method" is false. The standard measure of perf…

As far as I understand it, an advantage of the epsilon greedy algorithm is that it will relearn the best choice if it changes over time. Now, you could do that with a logarithmically-regretful algorithm as well, but it would take more time to relearn.

Re: 20 lines of code that beat A/B testing every time

#105

Post by micheljansen »

Although I can imagine this works very well for ecommerce websites and other things where there is a very obvious single measure of success, like for example:

  * the user clicked the button
  * the user signed up
  * the user put something in their cart
  * the user paid X
In this case, it's easy to create the feedback loop that is required for this testing method.

However, in the real world, things are not always that simple. What if you want to optimise:

  * the percentage of users that returns to the site 
  * the time that users spend on your site
  * the number of pages that they view in a session
I'm sure some of these metrics can also be plugged back into the bandit algorithm, but it's a lot more complicated.

Re: 20 lines of code that beat A/B testing every time

#107

Post by Sujan »

btilly wrote: post #93
pilom wrote: post #76

Earlier quoted context omitted.

"and don't give you reliable ways to know when to stop testing" Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing. In both multi-armed bandit and A/B testing you need to set ahead of time the number of users you are going to run your test against and stop the test at that point regardless of if your result is significant or not.

In theory, yes. In practice, no. See http://elem.com/~btilly/effective-ab-testing/index.html#asli... for part of a presentation that I did where I actually set up some reasonable fake tests, and ran simulations. What I found is that if there is a significant difference, the probability of coming to the wrong conclusion was (as you would expect) higher, but not that high before the underlying difference made mistakes…

Great presentation.

Re: 20 lines of code that beat A/B testing every time

#108

Post by conductrics »

timr wrote: post #72
vibrunazo wrote: post #70

Earlier quoted context omitted.

Why can't you do the same study from the bandit results? From what I understand, this the same as A/B testing, except it will only show a suboptimal result to 10% of the users instead of 50% (or more). After a few days of testing, can't you take a look at the statistics and analyze them the same way that you would for an A/B test? Then just stop testing? The A/B test just gives you the conversion rate of each option.…

"After a few days of testing, can't you take a look at the statistics and analyze them the same way that you would for an A/B test? Then just stop testing?" No, because the assumptions that underpin many statistical techniques are violated when you're not assigning people to cohorts consistently, and at random.

If you are using an epsilon-greedy approach (or something similar), then I believe that the data collected during the exploration portion - (the random calls) are open, albeit with less power due to reduced sample size, to standard hypothesis testing. Think of it this way, you might normally run your experiment on a subset of your traffic (population) - so only 20%, with the rest (80%) getting the current experience. With the e-greedy type of approach you are just swapping the 'current experience' with the maximum estimated experience, but that other 20% is still a random draw.

Re: 20 lines of code that beat A/B testing every time

#109

Post by its_so_on »

I always wondered why the great people I know can do so much better than having to do A/B testing in their own businesses. Sure they try new things, but they are most certainly not applying an A/B type algorithm.

It seems the article mentions something quite important in their algorithm: mostly do what has the most expected value. The people who are great just have a much better way to judge that. They can make a poster with 20 design choices (or 50 or 100) and make an estimate of what would probably work and what probably wouldn't on each one, from the size of poster, to the font sizes and types, whitespace, where to place different elements, graphics choices, etc etc etc. They certainly include a random aspect, but this is the exception rather than being the norm. Mostly what dictates choices is your expected returns on them, and the random aspects are compared with these.

They do pay exquisite attention to their random choices: "This week I decided to see what would happen if I mixed up the day, date, location, and description rather than have it be in logical order, to see if this engaged people any more" (or: to change any other choice randomly). But it is the exception rather than the norm, and done rarely rather than often. They still pay attention to the results, which helps inform their "expected value" function.

(I didn't spend much time on the article or the linked papers, so please feel free to correct me if I'm misinterpreting. A rigorous algorithm doesn't have much to do with real-world choices, and we are simply nowhere near having an automated web service to write your copy, regardless of how many users get to give feedback on it. So the whole thing isn't very interesting to me, and the above is just my impression of 'why'.)

Re: 20 lines of code that beat A/B testing every time

#110

Post by conductrics »

cbsmith wrote: post #98
noelwelsh wrote: post #94

First up, the sales pitch: we provide bandit optimisation SaaS at Myna: http://mynaweb.com Now, that's out of the way, let's discuss the article. I like the epsilon-greedy algorithm because it's simple to understand and implement, and easy to extend. However, to claim "The strategy that has been shown to win out time after time in practical problems is the epsilon-greedy method" is false. The standard measure of perf…

Yeah, I think the problem here is that trying to be a little bit smart kind of gets you in to the space where really you should be doing things a LOT smart. A/B testing provides data that doesn't require much in the way of brains to interpret and is hard to draw poor conclusions from (beyond treating something as statistically significant that is not). Once you step off in to epsilon-greedy, you fall in to the whole…

Actually, you kind of are already in the RL space when using AB testing to make online decisions, you just may not be thinking of it that way. From Sutton & Barto "Reinforcement learning is learning what to do--how to map situations to actions--so as to maximize a numerical reward signal." That is exactly what you are doing when applying A/B style hypothesis testing to inform decisions in an online application. Plus, personally, I think A/B testing is, in a way, much harder to interpret, at least most folks interpret wrong, which isn't a knock, since it is provides a non-intuitive - at least to me ;) - result.
Post reply on HN