Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

111–120 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#111

Post by zeroCalories »

sweezyjeezy wrote: post #104
ertdfgcvb wrote: post #85

Earlier quoted context omitted.

I don't follow. In this case would sampling 50/50 always give better/unbiased results on the experiment?

Sampling 50/50 will always give you the best chance of picking the best ultimate 'winner' in a fixed time horizon, at the cost of only sampling the winning variant 50% of the time. That's true if the reward rates are fixed or not. But some changes in reward rates will also cause MAB aggregate statistics to skew in a way that they shouldn't for a 50/50 split yeah.

What do you think of using the epsilon-first approach then? We could explore for that fixed time horizon, then start choosing greedy after that. I feel like the only downside is that adding new arms becomes more complicated.

Re: Lines of code that beat A/B testing (2012)

#112

Post by zeroCalories »

bartread wrote: post #76
blagie wrote: post #64

Earlier quoted context omitted.

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

Sorry but bubble sort is a terrible example here. You implement a more difficult sorting algorithm, like quicksort, because the benefits of doing so, versus using bubble sort, are in many cases huge. I.e., the juice is worth the squeeze. Whereas the comment you’re responding to is rightly pointing out that for most orgs, the marginal gains of using an approach more complex than Epsilon greedy probably aren’t worth it…

Bubble sort is a great example because it can out perform quicksort if the input is small.

Re: Lines of code that beat A/B testing (2012)

#113

Post by reader9274 »

MichaelDickens wrote: post #110

# for each lever, # calculate the expectation of reward. # This is the number of trials of the lever divided by the total reward # given by that lever. # choose the lever with the greatest expectation of reward. If I'm not mistaken, this pseudocode has a bug that will result in choosing the expected worst option rather than the expected best option. I believe it should read "total reward given by the lever divided by…

Correct, that's why I don't trust reading code comments

Re: Lines of code that beat A/B testing (2012)

#114

Post by [deleted] »

sweezyjeezy wrote: post #108
tjbai wrote: post #91

Earlier quoted context omitted.

I agree that there's an exploration-exploitation tradeoff, but for what you specifically suggest wouldn't you presumably just normalize by sample size? You wouldn't allocate based off total conversions, but rather a percentage.

Yes but here's a exaggerated version - say were to sample for a week at 50/50 when the base conversion rate was at 4%, then we sample at 25/75 for a week with the base conversion rate bumped up to 8% due to a sale. The average base rate for the first variant is 5.3%, the second is 6.4%. Generally the favoured variant's average will shift faster because we are sampling it more.

[deleted]

Re: Lines of code that beat A/B testing (2012)

#115

Post by recursivecaveat »

iforgot22 wrote: post #31

I don't like how this dismisses the old approach as "statistics are hard for most people to understand." This algo beats A/B testing in terms of maximizing how many visitors get the best feature. But is that really a big enough concern IRL that people are interested in optimizing it every time? Every little dynamic lever adds complexity to a system.

Indeed perhaps we should applaud people for choosing statistical tools that are relatively easy to use and interpret, rather than deride them for not stepping up to the lathe that they didn't really need and we admit has lots of sharp edges.

Re: Lines of code that beat A/B testing (2012)

#117

Post by jwr »

I've been using a Bernoulli bandit for many years now. Like the original author, I am not quite sure why more people don't use it — for all practical purposes, if you have stuff to do, it is superior to simple A/B testing in every way. The "set it and forget it" feature is really nice.

Another thing that I noticed while writing the code (and took advantage of) is that it is insanely scalable in a distributed system, using HyperLogLog and Bloom filters. I may or may not have totally over-engineered my code to take advantage of that, even though my site gets ridiculously low click numbers :-)

Re: Lines of code that beat A/B testing (2012)

#118

Post by petesergeant »

xp84 wrote: post #84
awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

> politically derisks the change, by tying it's deployment to rigorous testing that proves it at least does no harm to the existing process before applying it to all users. I just want to drop here the anecdata that I've worked for a total of about 10 years in startups that proudly call themselves "data-driven" and which worshipped "A/B testing." One of them hired a data science team which actually did some decently…

> Basically, I suspect a lot of small and medium companies say they do "A/B testing" and are "data-driven" when really they're just using slightly fancy feature flags and relying on some director's gut feelings.

see also Scrum and Agile. Or continuous deployment. Or anything else that's hard to do well, and easier to just cargo-cult some results on and call it done.

Re: Lines of code that beat A/B testing (2012)

#119

Post by necovek »

hamandcheese wrote: post #98
awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

I feel like you are trying to say "sometimes people just need a feature flag". Which is of course true.

A feature flag can still be fully on or fully off.

Why they might conflate A/B testing with gradual rollout is control over who gets the feature flag on and who doesn't.

In a sense, A/B testing is a variant of gradual rollout, where you've done it so you can see differences in feature "performance" (eg. funnel dashboards) vs just regular observability (app is not crashing yet).

Basically, a gradual rollout for the purposes of an A/B test.

Post reply on HN