Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

71–80 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#71

Post by s1mplicissimus »

kridsdale1 wrote: post #61
ivalm wrote: post #54

Earlier quoted context omitted.

You can assign multiarm bandit trials on a lazy per user basis. So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.

The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.

careful when doing that though! i've seen some big eyes when people assumed IDs to be uniform randomly distributed and suddenly their "test group" was 15% instead of the intended 1%. better generate a truely random value using your languages favorite crypto functions and be able to work with it without fear of busting production

Re: Lines of code that beat A/B testing (2012)

#72

Post by iforgot22 »

saintfire wrote: post #20
nottorp wrote: post #4

"People distrust things that they do not understand, and they especially distrust machine learning algorithms, even if they are simple." How times have changed :)

Just had to anthropomorphize machine learning.

More like, had to make it really good

Re: Lines of code that beat A/B testing (2012)

#73

Post by ryan-duve »

kridsdale1 wrote: post #61
ivalm wrote: post #54

Earlier quoted context omitted.

You can assign multiarm bandit trials on a lazy per user basis. So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.

The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.

How do you handle different users having different numbers of trials when calculating the "click through rate" described in the article?

Re: Lines of code that beat A/B testing (2012)

#74

Post by eru »

throwup238 wrote: post #67
awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

> In short, not many people want to funnel users through N code paths with slightly different behaviors, because not many people have a ton of users, a ton of engineering capacity, and a ton of potential upside from marginal improvements. I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. I figure by the law of probabilities they would have gotten a…

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results.

Well, at least it looks like they avoided p-hacking to show more significance than they had! That's ahead of much of science, alas.

Re: Lines of code that beat A/B testing (2012)

#75

Post by ryan-duve »

tracerbulletx wrote: post #8

A lot of sites don't have enough traffic to get statistical significance with this in a reasonable amount of time and it's almost always testing a feature more complicated than button color where you aren't going to have more than the control and variant.

> A lot of sites don't have enough traffic to get statistical significance with this in a reasonable amount of time

What's nice about AB testing is the decision can be made on point estimates, provided the two choices don't have different operational "costs". You don't need to know that A is better than B, you just need to pick one and the point estimate gives the best answer with the available data.

I don't know of a way to determine whether A is better than B with statistical significance without letting the experiment run, in practice, for way too long.

Re: Lines of code that beat A/B testing (2012)

#76

Post by bartread »

blagie wrote: post #64
timr wrote: post #11

Earlier quoted context omitted.

Just a warning to those people who are potentially implementing it: it doesn't really matter. The blog author addresses this, obliquely (says that the simplest thing is best most of the time), but doesn't make it explicit. In my experience, obsessing on the best decision strategy is the biggest honeypot for engineers implementing MAB. Epsilon-greedy is very easy to implement and you probably don't need anything more.…

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

Sorry but bubble sort is a terrible example here. You implement a more difficult sorting algorithm, like quicksort, because the benefits of doing so, versus using bubble sort, are in many cases huge. I.e., the juice is worth the squeeze.

Whereas the comment you’re responding to is rightly pointing out that for most orgs, the marginal gains of using an approach more complex than Epsilon greedy probably aren’t worth it. I.e., the juice isn’t worth the squeeze.

Re: Lines of code that beat A/B testing (2012)

#77

Post by ryan-duve »

IshKebab wrote: post #52

This is fine as long as your users don't mind your site randomly changing all the time.

This can be addressed with some variant of

    random.seed(hash(user_id))
I think the bigger problem is handling the fact that not all users click through the same number of times.

Re: Lines of code that beat A/B testing (2012)

#78

Post by desert_rue »

cle wrote: post #40
randomcatuser wrote: post #34

Earlier quoted context omitted.

Yeah basically. The idea is that somehow this is the data-optimal way of determining which one is the best (rather than splitting your data 50/50 and wasting a lot of samples when you already know) The caveats (perhaps not mentioned in the article) are: - Perhaps you have many metrics you need to track/analyze (CTR, conversion, rates on different metrics), so you can't strictly do bandit! - As someone mentioned below…

It still misses the biggest challenge though--defining "best", and ensuring you're actually measuring it and not something else. It's useful as long as your definition is good enough and your measurements and randomizations aren't biased. Are you monitoring this over time to ensure that it continues to hold? If you don't, you risk your MAB converging on something very different from what you would consider "the best"…

Are you saying that it may do something like improve click-the-button conversion but lead to less sales overall?

Re: Lines of code that beat A/B testing (2012)

#80

Post by sweezyjeezy »

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious.

Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a little more, say 60/40. You now start running a sale - and the conversion rate for both sides goes up equally. But since you are now sampling more from the treatment variant, its aggregate conversion rate goes up faster than the control - you start weighting even more towards that variant.

Fluctuating reward rates are everywhere in e-commerce, and tend to destabilise MAB proportions, even on two identical variants, they can even cause it to lean towards the wrong one. There are more sophisticated MAB approaches that try to remove the identical reward-rate assumption - they have to model a lot more uncertainty, and so optimise more conservatively.

Post reply on HN