Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lines of code that beat A/B testing (2012)

stevehanov.ca

Post reply on HN
Print view View original

151–160 of 180 posts

Re: Lines of code that beat A/B testing (2012)

#151

Post by timr »

blagie wrote: post #64
timr wrote: post #11

Earlier quoted context omitted.

Just a warning to those people who are potentially implementing it: it doesn't really matter. The blog author addresses this, obliquely (says that the simplest thing is best most of the time), but doesn't make it explicit. In my experience, obsessing on the best decision strategy is the biggest honeypot for engineers implementing MAB. Epsilon-greedy is very easy to implement and you probably don't need anything more.…

"Easy to implement" is a good reason to use bubble sort too. In a normal universe, you just import a different library, so both are the same amount of work to implement. Multiarmed bandit seems theoretically pretty, but it's rarely worth it. The complexity isn't the numerical algorithm but state management. * Most AB tests can be as simple as a client-side random() and a log file. * Multiarmed bandit means you need a…

You've either missed the point of what I wrote, or you're arguing with someone else.

I'm talking about the difference between epsilon-greedy vs. a more complex optimization scheme within the context of implementing MAB. You're making arguments about A/B testing vs MAB.

Re: Lines of code that beat A/B testing (2012)

#152

Post by hinkley »

hamandcheese wrote: post #99
rco8786 wrote: post #70

Earlier quoted context omitted.

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Yea, I've been here too. And in every analytics meeting everyone went "well, we know it's not statistically significant but we'll call it the winner anyway". Every. Single. Time. Such a waste of resources.

Is it a waste? You proved the change wasn't harmful.

You can still enshittify something by degrees this way.

I think the disconnect here is some people thinking A/B testing is something you try once a month, and someplace like Amazon where you do it all the time and with hundreds of employees poking things.

Re: Lines of code that beat A/B testing (2012)

#153

Post by hinkley »

Someone wrote: post #129
baxtr wrote: post #59

Earlier quoted context omitted.

I wouldn’t call that A/B testing but rather a gradual roll-out.

I think gradual rollout can use the same mechanism, but for a different readon: avoiding pushing out a potentially buggy product to all users in one sweep. It becomes an A/B test when you measure user activity to decide whether to roll out to more users.

Has my CPU use gone up? No.

Have my error logs gotten bigger? No.

Have my tech support calls gone up? No.

Okay then turn the dial farther.

Re: Lines of code that beat A/B testing (2012)

#154

Post by hinkley »

alex7o wrote: post #68
baxtr wrote: post #59

Earlier quoted context omitted.

I wouldn’t call that A/B testing but rather a gradual roll-out.

I think parent is confusing A/B testing with feature flags, which can be used for A/B tests but also for roll-outs.

Feature flags tend to be all or nothing and/or A/B testing instrumentation can be used to roll out feature flags.

It’s complicated.

Re: Lines of code that beat A/B testing (2012)

#155

Post by hinkley »

sweezyjeezy wrote: post #80

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…

Motivations can vary on a diurnal basis too. Or based on location. It means something different if I’m using homedepot.com at home or standing in an aisle at the store.

And with physical retailers with online catalogs, an online sale of one item may cannibalize an in-store purchase of not only that item but three other incidental purchases.

But at the end of the day your 60/40 example is just another way of saying: you don’t try to compare two fractions with a different denominator. It’s a rookie mistake.

Re: Lines of code that beat A/B testing (2012)

#156

Post by dang »

85392_school wrote: post #3

Previously discussed: https://news.ycombinator.com/item?id=11437114 https://news.ycombinator.com/item?id=4040022

Thanks! Macroexpanded:

20 lines of code that beat A/B testing (2012) - https://news.ycombinator.com/item?id=11437114 - April 2016 (157 comments)

20 lines of code that beat A/B testing every time - https://news.ycombinator.com/item?id=4040022 - May 2012 (147 comments)

Re: Lines of code that beat A/B testing (2012)

#157

Post by hinkley »

necovek wrote: post #123
sweezyjeezy wrote: post #80

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…

> ...the conversion rate for both sides goes up equally. If the conversion rate "goes up equally", why did you not measure this and use that as a basis for your decisions? > its aggregate conversion rate goes up faster than the control - you start weighting even more towards that variant. This sounds simply like using bad math. Wouldn't this kill most experiments that start with 10% for the variant that do not provid…

It is a universal truth that people fuck up statistical math.

    There are three kinds of lies: lies, damn lies, and statistics
If you aren’t testing at exactly 50/50 - and you can’t because my plan for visiting a site and for how long will never be equivalent to your plan, then any other factors that can affect conversion rate will cause one partition to go up faster than the other. You have to test at a level of Amazon to get statistical significance anyway.

And as many if us have told people until they’re blue in the face: we (you) are not a FAANG company and pretending to be one won’t work.

Re: Lines of code that beat A/B testing (2012)

#158

Post by hinkley »

zeroCalories wrote: post #111
sweezyjeezy wrote: post #104

Earlier quoted context omitted.

Sampling 50/50 will always give you the best chance of picking the best ultimate 'winner' in a fixed time horizon, at the cost of only sampling the winning variant 50% of the time. That's true if the reward rates are fixed or not. But some changes in reward rates will also cause MAB aggregate statistics to skew in a way that they shouldn't for a 50/50 split yeah.

What do you think of using the epsilon-first approach then? We could explore for that fixed time horizon, then start choosing greedy after that. I feel like the only downside is that adding new arms becomes more complicated.

What percent of companies using A/B testing do you think know what the Texas Sharpshooter is and how to identify it, let alone what epsilon is or what it means?

Re: Lines of code that beat A/B testing (2012)

#159

Post by hinkley »

jvans wrote: post #106
tjbai wrote: post #91

Earlier quoted context omitted.

I agree that there's an exploration-exploitation tradeoff, but for what you specifically suggest wouldn't you presumably just normalize by sample size? You wouldn't allocate based off total conversions, but rather a percentage.

Imagine a scenario where option B does 10x better than option A during the morning hours but -2x worse the rest of the day. If you start the multi armed bandit in the morning it could converge to option B quickly and dominate the rest of the day even though it performs worse then. Or in the above scenario option B performs a lot better than option A but only with the sale going, otherwise option B performs worse.

One of the problems we caught only once or twice: mobile versus desktop shifting with time of day, and what works on mobile may work worse than on desktop.

We weren’t at the level of hacking our users, just looking at changes that affect response time and resource utilizations, and figuring out why a change actually seems to have made things worse instead of better. It’s easy for people to misread graphs. Especially if the graphs are using Lying with Statistics anti patterns.

Re: Lines of code that beat A/B testing (2012)

#160

Post by ted_dunning »

necovek wrote: post #123
sweezyjeezy wrote: post #80

One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…

> ...the conversion rate for both sides goes up equally. If the conversion rate "goes up equally", why did you not measure this and use that as a basis for your decisions? > its aggregate conversion rate goes up faster than the control - you start weighting even more towards that variant. This sounds simply like using bad math. Wouldn't this kill most experiments that start with 10% for the variant that do not provid…

No. This isn't just bad math.

The problem here is that the weighting of the alternatives changes over time and the thing you are measuring may also change. If you start by measuring the better option, but then bring in the worse option in a better general climate, you could easily conclude the worse option is better.

To give a concrete example, suppose you have two versions of your website, one in English and one in Japanese. Worldwide, Japanese speakers tend to be awake at different hours than English speakers. If you don't run your tests over full days, you may bias the results to one audience or the other. Even worse, weekend visitors may be much different than weekday visitors so you may need to slow down to full weeks for your tests.

Changing tests slowly may mean that you can only run a few tests unless you are looking at large effects which will show through the confounding effects.

And that leads back to the most prominent normal use which is progressive deployments. The goal there is to test whether the new version is catastrophically worse than the old one so that as soon as you have error bars that bound the new performance away from catastrophe, you are good to go.

Post reply on HN