Run fewer, better A/B tests • Hacker News Board

Re: Run fewer, better A/B tests

#21

Post by eximius » Sun, Jun 27, 2021, 1:19 AM UTC

gingerlime wrote: post #16

eximius wrote: post #3

I'm going to stick with multiarm bandit testing.

What tools/frameworks are you using for running and analysing results?

At my previous job when it was relevant, I wrote something in house.

A higher order component would pick which variant at runtime (cause we had problems with SSR, or that would be more appropriate). Cached the picks in cookies.

In house charting and probability calculations to determine what P(X>Y) was for each experiment pair. Then we'd just manually prune them occasionally (since the bad ones weren't being displayed, timeliness didn't much matter). Periodically re-introduce old variants by hand if we thought it was worth it.

Re: Run fewer, better A/B tests

#22

Post by kqr » Sun, Jun 27, 2021, 4:21 AM UTC

jonathankoren wrote: post #20

uyt wrote: post #14

Earlier quoted context omitted.

I think it's known as the "novelty effect" in the industry.

Yup. A common way to handle this is to throw away the earliest responses, and then run your analysis. This is sometimes called “burn in”. If the lift goes away, you know it wasn’t real.

Or compare only new users that have only ever seen either the control or the treatment.

But the effect is real, sure enough. Only to get at it you have to mix up the title/subject line every now and then.

Re: Run fewer, better A/B tests

#23

Post by jonathankoren » Sun, Jun 27, 2021, 5:40 AM UTC

kqr wrote: post #22

jonathankoren wrote: post #20

Earlier quoted context omitted.

Yup. A common way to handle this is to throw away the earliest responses, and then run your analysis. This is sometimes called “burn in”. If the lift goes away, you know it wasn’t real.

Or compare only new users that have only ever seen either the control or the treatment. But the effect is real, sure enough. Only to get at it you have to mix up the title/subject line every now and then.

I wouldn’t do that.

First, it seems impractical because you need a high user growth to be able to get enough samples in a reasonable time to run your experiment to statistical significance. If you are seeing high growth, how many of those accounts are real people rather than bots or duplicate accounts? Maybe that isn’t too big of a deal for an email campaign, but for apps and websites, it matters, because it of the next reason.

Secondly, there’s no reason to believe that new users behave like old users, in fact there are probably many reasons to believe that they do not. Because your A/B population isn’t representative of the actual population of users, there’s no reason to believe whatever effect you measured on the neophyte users is going to carry over to the experienced users. Maybe, but it’s not guaranteed.

Random sampling of users is better, because the A/B population is drawn from the general population, and so the only confounding property is the novelty effect. Doing what you suggest eliminates the novelty effect, but now you can’t generalize your learnings at all.