Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

51–60 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#51

Post by iforgot22 »

rerdavies wrote: post #49
iforgot22 wrote: post #31

I don't like how this dismisses the old approach as "statistics are hard for most people to understand." This algo beats A/B testing in terms of maximizing how many visitors get the best feature. But is that really a big enough concern IRL that people are interested in optimizing it every time? Every little dynamic lever adds complexity to a system.

I think you missed the point. It's not about which visitors get the best feature. It's about how to get people to PUSH THE BUTTON!!!!! Which is kind of the opposite of the best feature. The goal is to make people do something they don't want to do. Figuring out best features is a completely different problem.

I didn't say it was the best for the user. Really the article misses this by comparing a new UI feature to a life-saving drug, but it doesn't matter. The point is, whatever metric you're targeting, do you use this algo or fixed group sizes?

Re: Lines of code that beat A/B testing (2012)

#54

Post by ivalm »

taion wrote: post #50

The problem with this approach is that it requires the system doing randomization to be aware of the rewards. That doesn't make a lot of sense architecturally – the rewards you care about often relate to how the user engages with your product, and you would generally expect those to be collected via some offline analytics system that is disjoint from your online serving system. Additionally, doing randomization on a…

You can assign multiarm bandit trials on a lazy per user basis.

So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.

Re: Lines of code that beat A/B testing (2012)

#56

Post by atombender »

fitsumbelay wrote: post #55

I've only read the first paragraph so bear with me but I'm not understanding the reasoning behind "A/B testing drugs is bad because only half of the sample can potentially benefit" when the whole point is to delineate the gots and got-nots ...

If the drug is effective and safe, then one half of the patients lost out on the benefit. You are intentionally "sacrificing" the control arm.

(Of course, the whole point is that the benefit and safety are not certain, so I think the term "sacrifice" used in the article is misleading.)

Re: Lines of code that beat A/B testing (2012)

#57

Post by m3kw9 »

If you only keep your entire site static while test one variable change at a time, it could be statistically significant, other wise if your flow changes some where while you do this algo, it may be misleading you into a color and then under perform because you've made a change else where before users get to this page.

Re: Lines of code that beat A/B testing (2012)

#58

Post by cauch »

LPisGood wrote: post #45
cauch wrote: post #42

Earlier quoted context omitted.

Another way of seeing the situation: let run your MAB solution for a while. Orange has been tested 17 times and blue has been tested 12 times. This is exactly equivalent of doing a A/B testing where you display 1 time the orange button to 17 persons and 1 time the blue button to 12 persons. The trick is to find the exact best number of test for each color so that we have good statistical significance. MAB does not do…

> The trick is to find the exact best number of test for each color so that we have good statistical significance Yes, A/B testing will force through enough trials to get statistical significance(it is definitely a “exploration first strategy), but in many cases, you care about maximizing reward as well, in particular during testing. A/B testing does very poorly at balancing exploitation with exploitation in general.…

But the proposed MAB system does not even propose a method to know when this system needs to be stopped (and remove all the choices except the best one).

With the A/B testing, you can do power analysis whenever you want, including in the middle of the experiment. It will just be an iterative adjustment that converges.

In fact, you can even run on all possibilities in advance (if A get 1% and B get 1%, how many A and B do I need, if A get 2% and B get 1%, if A get 3% and B get 1%, ...) and it will give you the exact boundaries to stop for any configurations before even running the experiment. You will just have to stop trialing option A as soon as option A crosses the already decided significance threshold for A.

So, no, the A/B testing will never run forever. And A/B testing will always be better than the MAB solution, because you will have a better way to stop trying a bad solution as soon as you have crossed the threshold you decided is enough to consider it's a bad solution.

Re: Lines of code that beat A/B testing (2012)

#59

Post by baxtr »

ljm wrote: post #48
awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

Tracks that I’ve primarily seen A/B tests used as a mechanism for gradual rollout rather than pure data-driven experimentation. Basically expose functionality to internal users by default then slowly expand it outwards to early adopters and then increment it to 100% for GA. It’s helpful in continuous delivery setups since you can test and deploy the functionality and move the bottleneck for releasing beyond that.

I wouldn’t call that A/B testing but rather a gradual roll-out.

Re: Lines of code that beat A/B testing (2012)

#60

Post by kridsdale1 »

tracerbulletx wrote: post #8

A lot of sites don't have enough traffic to get statistical significance with this in a reasonable amount of time and it's almost always testing a feature more complicated than button color where you aren't going to have more than the control and variant.

I’ve only implemented A/B/C tests at Facebook and Google, with hundreds of millions of DAU on the surfaces in question, and three groups is still often enough to dilute the measurement in question below stat-sig.
Post reply on HN