Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

91–100 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#91

Post by tjbai »

sweezyjeezy wrote: post #80

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…

I agree that there's an exploration-exploitation tradeoff, but for what you specifically suggest wouldn't you presumably just normalize by sample size? You wouldn't allocate based off total conversions, but rather a percentage.

Re: Lines of code that beat A/B testing (2012)

#92

Post by nine_k »

alex7o wrote: post #68
baxtr wrote: post #59

Earlier quoted context omitted.

I wouldn’t call that A/B testing but rather a gradual roll-out.

I think parent is confusing A/B testing with feature flags, which can be used for A/B tests but also for roll-outs.

Not the parent but some actual practitioners. A change is based on the gut feeling, and it's usually correct, but the internal politics require to demonstrate impartiality, so an "A/B test" is run, to show that the change is "objectively better", whether statistics show that or not.

Re: Lines of code that beat A/B testing (2012)

#94

Post by codazoda »

bartread wrote: post #79

Interesting post - certainly I can see myself wanting to tinker with Epsilon greedy - but the comments at the bottom are pretty off the chain, and many not in a good way. No auth commenting is certainly a brave decision in 2024.

I wish I could make this work without the mess though. Maybe an AI moderator could throw some of these away. Not that we need AI for everything, but I don’t have time to edit comments on my blog.

Re: Lines of code that beat A/B testing (2012)

#96

Post by np_tedious »

s1mplicissimus wrote: post #71
kridsdale1 wrote: post #61

Earlier quoted context omitted.

The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.

careful when doing that though! i've seen some big eyes when people assumed IDs to be uniform randomly distributed and suddenly their "test group" was 15% instead of the intended 1%. better generate a truely random value using your languages favorite crypto functions and be able to work with it without fear of busting production

The user ID is non uniform after hash and mod? How?

Re: Lines of code that beat A/B testing (2012)

#97

Post by LPisGood »

ertdfgcvb wrote: post #88
LPisGood wrote: post #24

Earlier quoted context omitted.

Multi-arm bandit does beat A/B testing in the sense that standard A/B testing does not seek to maximize reward during the testing period, MAB does. MAB also generalizes better to testing many things than A/B testing.

Isn't that the point of testing (to not maximize reward but rather wait and collect data)? It sounds like maximizing reward during the experiment period can bias the results

The great thing is that you can do both.

Re: Lines of code that beat A/B testing (2012)

#98

Post by hamandcheese »

awkward wrote: post #47

Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…

I feel like you are trying to say "sometimes people just need a feature flag". Which is of course true.

Re: Lines of code that beat A/B testing (2012)

#99

Post by hamandcheese »

rco8786 wrote: post #70
throwup238 wrote: post #67

Earlier quoted context omitted.

> In short, not many people want to funnel users through N code paths with slightly different behaviors, because not many people have a ton of users, a ton of engineering capacity, and a ton of potential upside from marginal improvements. I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. I figure by the law of probabilities they would have gotten a…

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Yea, I've been here too. And in every analytics meeting everyone went "well, we know it's not statistically significant but we'll call it the winner anyway". Every. Single. Time. Such a waste of resources.

Is it a waste? You proved the change wasn't harmful.

Re: Lines of code that beat A/B testing (2012)

#100

Post by ivalm »

kridsdale1 wrote: post #61
ivalm wrote: post #54

Earlier quoted context omitted.

You can assign multiarm bandit trials on a lazy per user basis. So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.

The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.

To make sure user id U doesn’t always end up in eg control group it’s useful to concatenate the id with experiment uuid.
Post reply on HN