Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

171–180 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#171

Post by lern_too_spel »

np_tedious wrote: post #168
lern_too_spel wrote: post #103

Earlier quoted context omitted.

If you mod by anything other than a power of two, it won't be. https://lemire.me/blog/2019/06/06/nearly-divisionless-random...

That article is mostly about speed. The following seems like the one thing that might be relevant: > Naively, you could take the random integer and compute the remainder of the division by the size of the interval. It works because the remainder of the division by D is always smaller than D. Yet it introduces a statistical bias That's all it says. Is the point here just that 2^31 % 17 is not zero, so 1,2,3 are potent…

> If so, this is not terribly important

It is not uniformly random, which is the whole point.

> That article is mostly about speed

The article is about how to actually achieve uniform random at high speed. Just doing mod is faster but does not satisfy the uniform random requirement.

Re: Lines of code that beat A/B testing (2012)

#172

Post by crazygringo »

LPisGood wrote: post #44
crazygringo wrote: post #38

Earlier quoted context omitted.

I'm not talking about the literature -- I'm talking about the extremely simplistic and sub-optimal procedure described in the post. If you want to get sophisticated, MAB properly done is essentially just A/B testing with optimal strategies for deciding when to end individual A/B tests, or balancing tests optimally for a limited number of trials. But again, it doesn't "beat" A/B testing -- it is A/B testing in that se…

I am talking about the literature which solves MAB in a variety of ways, including the one in the post. > MAB properly done is essentially just A/B testing Words are only useful insofar as their meanings invoke ideas, and in my experience absolutely no one thinks of other MAB strategies when someone talks about A/B testing. Sure, you can classify A/B testing as one extremely suboptimal approach to solving MAB problem…

> Sure, you can classify A/B testing as one extremely suboptimal approach to solving MAB problem. This classification doesn’t help much though, because the other MAB techniques do “magically increase the rewards” compared this simple technique.

You are quite simply wrong. There is nothing suboptimal about an A/B test between two choices performed until desired statistical significance. There is nothing you can do to magically increase anything.

If you think there is, you'll have to describe something specific. Because nowhere in the academic MAB literature does anyone attempt to state the contrary. And which, again, is why this blog post is so flawed.

Re: Lines of code that beat A/B testing (2012)

#173

Post by s1mplicissimus »

np_tedious wrote: post #96
s1mplicissimus wrote: post #71

Earlier quoted context omitted.

careful when doing that though! i've seen some big eyes when people assumed IDs to be uniform randomly distributed and suddenly their "test group" was 15% instead of the intended 1%. better generate a truely random value using your languages favorite crypto functions and be able to work with it without fear of busting production

The user ID is non uniform after hash and mod? How?

additional to the other excellent comments they will become non-uniform once you start deleting records. that will break all hopes you might have had in modulo and percentages being reliable partitions because the "holes" in your ID space could be maximally bad for whatever usecase you thought up.

Re: Lines of code that beat A/B testing (2012)

#174

Post by orasis »

timr wrote: post #11
isoprophlex wrote: post #5

As one of the comments below the article states, the probabilistic alternative to epsilon-greedy is worth exploring ad well. Take the "bayesian bandit", which is not much more complex but a lot more powerful. If you crave more bandits: https://jamesrledoux.com/algorithms/bandit-algorithms-epsilo...

Just a warning to those people who are potentially implementing it: it doesn't really matter. The blog author addresses this, obliquely (says that the simplest thing is best most of the time), but doesn't make it explicit. In my experience, obsessing on the best decision strategy is the biggest honeypot for engineers implementing MAB. Epsilon-greedy is very easy to implement and you probably don't need anything more.…

Thompson Sampling is trivial to implement, especially with binary rewards. ChatGPT can do it reliably from scratch.

Re: Lines of code that beat A/B testing (2012)

#175

Post by orasis »

ted_dunning wrote: post #163
blagie wrote: post #64

Earlier quoted context omitted.

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

Multi-armed bandit approaches do not imply an immediate feedback loop. They do the best you can do with delayed feedback or with episodic adjustment as well. So if you are doing A/B tests, it is quite reasonable to use Thompson sampling at fixed intervals to adjust the proportions. If your response variable is not time invariant, this is actually best practice.

Having significant experience with bandits in production, I strongly recommend only using them for immediate feedback. If the rewards are at all disconnected from the action you likely won’t be happy with the results.

Re: Lines of code that beat A/B testing (2012)

#176

Post by orasis »

jwr wrote: post #117

I've been using a Bernoulli bandit for many years now. Like the original author, I am not quite sure why more people don't use it — for all practical purposes, if you have stuff to do, it is superior to simple A/B testing in every way. The "set it and forget it" feature is really nice. Another thing that I noticed while writing the code (and took advantage of) is that it is insanely scalable in a distributed system,…

lol. My first MAB implementation also used HyperLogLog for tracking unique conversions. We over engineer alike it seems.

Re: Lines of code that beat A/B testing (2012)

#177

Post by orasis »

sarpdag wrote: post #135

I really like multi armed bandit approach, but struggles with common scenarios involving delayed rewards or multiple success criteria, such as testing ecommerce search with number of orders and GMV guardrails. For simple, immediate-feedback cases like button clicks, the specific implementation becomes less critical.

It’s best for immediate rewards. If you have delayed rewards there is a paper on sampling from the “delay distribution” that solves this.

Re: Lines of code that beat A/B testing (2012)

#178

Post by blagie »

timr wrote: post #151
blagie wrote: post #64

Earlier quoted context omitted.

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

You've either missed the point of what I wrote, or you're arguing with someone else. I'm talking about the difference between epsilon-greedy vs. a more complex optimization scheme within the context of implementing MAB. You're making arguments about A/B testing vs MAB.

One of us definitely misread something.

Let me spell out what I wrote for you:

* In most systems, the cost of implementing MAB >> AB

* The cost if implementing a more complex optimization scheme is only marginally higher than the cost of MAB

Therefore, if you're already going up to MAB, you might as well go all the way.

Re: Lines of code that beat A/B testing (2012)

#179

Post by np_tedious »

lern_too_spel wrote: post #171
np_tedious wrote: post #168

Earlier quoted context omitted.

That article is mostly about speed. The following seems like the one thing that might be relevant: > Naively, you could take the random integer and compute the remainder of the division by the size of the interval. It works because the remainder of the division by D is always smaller than D. Yet it introduces a statistical bias That's all it says. Is the point here just that 2^31 % 17 is not zero, so 1,2,3 are potent…

> If so, this is not terribly important It is not uniformly random, which is the whole point. > That article is mostly about speed The article is about how to actually achieve uniform random at high speed. Just doing mod is faster but does not satisfy the uniform random requirement.

If your number of AB testing combos cohorts is fewer then 100 then yeah this passes for being uniform

Re: Lines of code that beat A/B testing (2012)

#180

Post by lern_too_spel »

np_tedious wrote: post #179
lern_too_spel wrote: post #171

Earlier quoted context omitted.

> If so, this is not terribly important It is not uniformly random, which is the whole point. > That article is mostly about speed The article is about how to actually achieve uniform random at high speed. Just doing mod is faster but does not satisfy the uniform random requirement.

If your number of AB testing combos cohorts is fewer then 100 then yeah this passes for being uniform

It doesn't, mathematically. It might be good enough for some cases, but it is not good enough for cases that actually require uniformity.
Post reply on HN