Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

81–90 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#81

Post by kavenkanum »

awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

Derisking changes may not work sometimes. For example I don't use Spotify anymore, because of their ridiculous Ab tests. In one month I saw 3 totally different designs of the home and my fav playlists page on my Android phone. That's it. When you open Spotify only when you start your car then it's ridiculous that you can't find anything and you are in a hurry. That was it. I am no longer subscriber and a user of this shit service. Sometimes these tests are actually harmful. Maybe others are just driving and trying to manage Spotify at the same time and then we have actual killed people because of this. Harmless Indeed.

Re: Lines of code that beat A/B testing (2012)

#82

Post by groby_b »

That's a statistically valid approach. Technically correct, the best kind of correct.

Meanwhile, if your users get presented a different button whenever they come by, because the MAB is still pursuing its hill climbing, they'll rightfully accuse you of having extremely crappy UX. (And, sure, you can have MAB with user stickiness, but now you do need to talk about sampling bias)

And MAB hill climb doesn't work at all if you want to measure the long-term reward of a variation. You have no idea if the orange button has long-term retention impact. There are sure situations where you'd like to know.

Yes, it's a neat technique to have in your repertoire, but like any given technique, it's not the answer "every time".

Re: Lines of code that beat A/B testing (2012)

#83

Post by dkarl »

throwup238 wrote: post #67
awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

> In short, not many people want to funnel users through N code paths with slightly different behaviors, because not many people have a ton of users, a ton of engineering capacity, and a ton of potential upside from marginal improvements. I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. I figure by the law of probabilities they would have gotten a…

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results.

What I've seen in practice is that some places trust their designers' decisions and only deploy A/B tests when competent people disagree, or there's no clear, sound reason to choose one design over another. Surprise surprise, those alternatives almost always test very close to each other!

Other places remove virtually all friction from A/B testing and then use it religiously for every pixel in their product, and they get results, but often it's things like "we discovered that pink doesn't work as well as red for a warning button," stuff they never would have tried if they didn't have to feed the A/B machine.

From all the evidence I've seen in places I've worked, the motivating stories of "we increased revenue 10% by a random change nobody thought would help" may only exist in blog posts.

Re: Lines of code that beat A/B testing (2012)

#84

Post by xp84 »

awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

> politically derisks the change, by tying it's deployment to rigorous testing that proves it at least does no harm to the existing process before applying it to all users.

I just want to drop here the anecdata that I've worked for a total of about 10 years in startups that proudly call themselves "data-driven" and which worshipped "A/B testing." One of them hired a data science team which actually did some decently rigorous analysis on our tests and advised things like when we had achieved statistical significance, how many impressions we needed to have, etc. The other did not and just had someone looking at very simple comparisons in Optimizely.

In both cases, the influential management people who ultimately owned the decisions would simply rig every "test" to fit the story they already believed, by doing things like running the test until the results looked "positive" but not until it was statistically significant. Or, by measuring several metrics and deciding later on to make the decision based on whichever one was positive [at the time]. Or, by skipping testing entirely and saying we'd just "used a pre/post comparison" to prove it out. Or even by just dismissing a 'failure,' saying we would do it anyway because it's foundational to X, Y, and Z which really will improve (insert metric) The funny part is that none of these people thought they were playing dirty, they believed that they were making their decisions scientifically!

Basically, I suspect a lot of small and medium companies say they do "A/B testing" and are "data-driven" when really they're just using slightly fancy feature flags and relying on some director's gut feelings.

Re: Lines of code that beat A/B testing (2012)

#85

Post by ertdfgcvb »

sweezyjeezy wrote: post #80

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…

I don't follow. In this case would sampling 50/50 always give better/unbiased results on the experiment?

Re: Lines of code that beat A/B testing (2012)

#86

Post by mikepurvis »

xp84 wrote: post #84
awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

> politically derisks the change, by tying it's deployment to rigorous testing that proves it at least does no harm to the existing process before applying it to all users. I just want to drop here the anecdata that I've worked for a total of about 10 years in startups that proudly call themselves "data-driven" and which worshipped "A/B testing." One of them hired a data science team which actually did some decently…

At a small enough scale, gut feelings can be totally reasonable; taste is important and I'd rather follow an opinionated leader with good taste than someone who sits on their hands waiting for "the data". Anyway, your investors want you to move quickly because they're A/B testing you for surviveability against everything else in their portfolio.

The worst is surely when management make the investments in rigor but then still ignores the guidance and goes with their gut feelings that were available all along.

Re: Lines of code that beat A/B testing (2012)

#88

Post by ertdfgcvb »

LPisGood wrote: post #24
crazygringo wrote: post #16

No, multi-armed bandit doesn't "beat" A/B testing, nor does it beat it "every time". Statistical significance is statistical significance, end of story. If you want to show that option B is better than A, then you need to test B enough times. It doesn't matter if you test it half the time (in the simplest A/B) or 10% of the time (as suggested in the article). If you do it 10% of the time, it's just going to take you…

Multi-arm bandit does beat A/B testing in the sense that standard A/B testing does not seek to maximize reward during the testing period, MAB does. MAB also generalizes better to testing many things than A/B testing.

Isn't that the point of testing (to not maximize reward but rather wait and collect data)? It sounds like maximizing reward during the experiment period can bias the results

Re: Lines of code that beat A/B testing (2012)

#89

Post by tjbai »

ertdfgcvb wrote: post #87

I'm a layman. Isn't MAB changing the experiment parameters while the experiment is still running? That sounds like an easy way towards biased experiment results

From a purely technical definition of bias (difference in expected value of the estimator and the true value), MAB is not biased because "changing the experiment parameters" is just dynamically allocating a different sample size to each of the estimators, so the estimator still converges to the correct value.

You are correct that this setup can potentially mislead you, but this is because you might end up getting estimators with high variance. So, you might mistakenly see some early promising results for experiment group A and greedily assign all the requests to that group, even though it is not guaranteed that A is actually better than B.

This is the famous exploration-exploitation dilemma—should you maximize conversions by diverting everyone to group A or still try to collect more data from group B?

Re: Lines of code that beat A/B testing (2012)

#90

Post by awkward »

alex7o wrote: post #68
baxtr wrote: post #59

Earlier quoted context omitted.

I wouldn’t call that A/B testing but rather a gradual roll-out.

I think parent is confusing A/B testing with feature flags, which can be used for A/B tests but also for roll-outs.

I’m aware of the distinction. A/B testing is the killer app for feature flags from the perspective of business decision makers.
Post reply on HN