Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

161–170 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#161

Post by hinkley »

hruk wrote: post #102
taion wrote: post #50

The problem with this approach is that it requires the system doing randomization to be aware of the rewards. That doesn't make a lot of sense architecturally – the rewards you care about often relate to how the user engages with your product, and you would generally expect those to be collected via some offline analytics system that is disjoint from your online serving system. Additionally, doing randomization on a…

I don't know, all of these are pretty surmountable. We've done dynamic pricing with contextual multi-armed bandits, in which each context gets a single decision per time block and gross profit is summed up at the end of each block and used to reward the agent. That being said, I agree that MABs are poor for experimentation (they produce biased estimates that depend on somewhat hard-to-quantify properties of your poli…

You do know Amazon got sued and lost for showing different prices to different users? That kind of price discrimination is illegal in the US. Related to actual discrimination.

I think Uber gets away with it because it’s time and location based, not person based. Of course if someone starts pointing out that segregation by neighborhoods is still a thing, they might lose their shiny toys.

Re: Lines of code that beat A/B testing (2012)

#162

Post by hinkley »

kridsdale1 wrote: post #61
ivalm wrote: post #54

Earlier quoted context omitted.

You can assign multiarm bandit trials on a lazy per user basis. So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.

The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.

Just make sure you do the hash right so you don’t end up with cursed user IDs like EverQuest.

Re: Lines of code that beat A/B testing (2012)

#163

Post by ted_dunning »

blagie wrote: post #64
timr wrote: post #11

Earlier quoted context omitted.

Just a warning to those people who are potentially implementing it: it doesn't really matter. The blog author addresses this, obliquely (says that the simplest thing is best most of the time), but doesn't make it explicit. In my experience, obsessing on the best decision strategy is the biggest honeypot for engineers implementing MAB. Epsilon-greedy is very easy to implement and you probably don't need anything more.…

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

Multi-armed bandit approaches do not imply an immediate feedback loop. They do the best you can do with delayed feedback or with episodic adjustment as well.

So if you are doing A/B tests, it is quite reasonable to use Thompson sampling at fixed intervals to adjust the proportions. If your response variable is not time invariant, this is actually best practice.

Re: Lines of code that beat A/B testing (2012)

#164

Post by SideQuark »

What an odd, self-contradictory post. "In recent years, hundreds of the brightest minds of modern civilization have been hard at work not curing cancer.", along with phrases like "defective by design" and implied loss by not giving all people in a drug trial the new medicine, imply he thinks all of a thing needs allocated to what he perceives (incorrectly, trivially, shown below) as the best use. Then he states in the multiarm bandit to waste (from his multiple other statements) 10% of what is the best use on random other uses for exploration.

However all this fails. For optimal output (be it drug research, allocation of brains, how to run a life), putting all resources on the problem/thing that is the "most important" is sub-optimal use of resources. It's always better expected return to allocate resources to where that spent resource has the best return. If that place is apps, not cancer, then wishing for brains to work on cancer because some would view that as a more important problem may simply be a waste of brains.

So if cancer is going to be incredibly hard to solve, and mankind empirically gets utility from better apps, then a better use is to put those brains on apps - then they're not wasted on an probably not solvable problem and are put to use making things that do increase value.

He also ignores that in real life the cost to have a zillion running experiments constantly flipping alternatives does not scale, so in no way can a company at scale replace A/B with multiarm bandits. One reason is simple: at any time a large company is running 1000s to maybe 100k A/B tests, each running maybe 6 months, at which point code path is selected, dead paths removed, and this repeats continually.If that old code is not killed, and every feature from all time needs to be on/off randomly, then there is no way over time to move much of the app forwards. It's not effective or feasibly to build many new features if you must also allow interacting with those from 5-10 years ago.

A simple google shows tons more reasons, form math to practical, that this post is bad advice.

Re: Lines of code that beat A/B testing (2012)

#165

Post by necovek »

ted_dunning wrote: post #160
necovek wrote: post #123

Earlier quoted context omitted.

> ...the conversion rate for both sides goes up equally. If the conversion rate "goes up equally", why did you not measure this and use that as a basis for your decisions? > its aggregate conversion rate goes up faster than the control - you start weighting even more towards that variant. This sounds simply like using bad math. Wouldn't this kill most experiments that start with 10% for the variant that do not provid…

No. This isn't just bad math. The problem here is that the weighting of the alternatives changes over time and the thing you are measuring may also change. If you start by measuring the better option, but then bring in the worse option in a better general climate, you could easily conclude the worse option is better. To give a concrete example, suppose you have two versions of your website, one in English and one in…

I mean, sure you could test over only part of the day, but if you do, that is, imho, bad math.

Eg. I could sum up 10 (decimal) and 010 (octal) as 20, but because they were the same digits in different numbering systems, you need to normalize the values first to the same base.

Or I could add up 5 GBP, 5 USD, 5 EUR and 5 JPY and claim I got 20 of "currency", but it doesn't really mean anything.

Otherwise, we are comparing incomparable values, and that's bad math.

Sure, percentages is what everybody gets wrong (hey percentage points vs percentage), but that does not make them not wrong. And knowing what is comparable when you simply talk in percentages, even more so (as per your examples).

Re: Lines of code that beat A/B testing (2012)

#166

Post by sweezyjeezy »

necovek wrote: post #136
sweezyjeezy wrote: post #108

Earlier quoted context omitted.

Yes but here's a exaggerated version - say were to sample for a week at 50/50 when the base conversion rate was at 4%, then we sample at 25/75 for a week with the base conversion rate bumped up to 8% due to a sale. The average base rate for the first variant is 5.3%, the second is 6.4%. Generally the favoured variant's average will shift faster because we are sampling it more.

Uhm, this still sounds like just bad math. While it's non-obvious this is the effect, anyone analyzing the results should be aware of it and should only compare weighted averages, or per distinct time periods. And therein is the largest problem with A/B testing: it's mostly done by people not understanding the math subtleties, thus they will misinterpret results in either direction.

Agreed, and articles like this don't help. That's the only point I was trying to make really.

Re: Lines of code that beat A/B testing (2012)

#167

Post by taion »

hruk wrote: post #102
taion wrote: post #50

The problem with this approach is that it requires the system doing randomization to be aware of the rewards. That doesn't make a lot of sense architecturally – the rewards you care about often relate to how the user engages with your product, and you would generally expect those to be collected via some offline analytics system that is disjoint from your online serving system. Additionally, doing randomization on a…

I don't know, all of these are pretty surmountable. We've done dynamic pricing with contextual multi-armed bandits, in which each context gets a single decision per time block and gross profit is summed up at the end of each block and used to reward the agent. That being said, I agree that MABs are poor for experimentation (they produce biased estimates that depend on somewhat hard-to-quantify properties of your poli…

You can do that, but now you have a runtime dependency on your analytics system, right? This can be reasonable for a one-off experimentation system but it's not likely you'll be able to do all of your experimentation this way.

Re: Lines of code that beat A/B testing (2012)

#168

Post by np_tedious »

lern_too_spel wrote: post #103
np_tedious wrote: post #96

Earlier quoted context omitted.

The user ID is non uniform after hash and mod? How?

If you mod by anything other than a power of two, it won't be. https://lemire.me/blog/2019/06/06/nearly-divisionless-random...

That article is mostly about speed. The following seems like the one thing that might be relevant:

> Naively, you could take the random integer and compute the remainder of the division by the size of the interval. It works because the remainder of the division by D is always smaller than D. Yet it introduces a statistical bias

That's all it says. Is the point here just that 2^31 % 17 is not zero, so 1,2,3 are potentially happening slightly more than 15,16? If so, this is not terribly important

Re: Lines of code that beat A/B testing (2012)

#169

Post by hruk »

hinkley wrote: post #161
hruk wrote: post #102

Earlier quoted context omitted.

I don't know, all of these are pretty surmountable. We've done dynamic pricing with contextual multi-armed bandits, in which each context gets a single decision per time block and gross profit is summed up at the end of each block and used to reward the agent. That being said, I agree that MABs are poor for experimentation (they produce biased estimates that depend on somewhat hard-to-quantify properties of your poli…

You do know Amazon got sued and lost for showing different prices to different users? That kind of price discrimination is illegal in the US. Related to actual discrimination. I think Uber gets away with it because it’s time and location based, not person based. Of course if someone starts pointing out that segregation by neighborhoods is still a thing, they might lose their shiny toys.

Indeed, we are well aware.

Re: Lines of code that beat A/B testing (2012)

#170

Post by hruk »

taion wrote: post #167
hruk wrote: post #102

Earlier quoted context omitted.

I don't know, all of these are pretty surmountable. We've done dynamic pricing with contextual multi-armed bandits, in which each context gets a single decision per time block and gross profit is summed up at the end of each block and used to reward the agent. That being said, I agree that MABs are poor for experimentation (they produce biased estimates that depend on somewhat hard-to-quantify properties of your poli…

You can do that, but now you have a runtime dependency on your analytics system, right? This can be reasonable for a one-off experimentation system but it's not likely you'll be able to do all of your experimentation this way.

No, you definitely have to pick your battles. Something that you want to continuously optimize over time makes a lot more sense than something where it's reasonable to test and the commit to a path forever.
Post reply on HN