Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical significance on a shoestring budget

alexeymk.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Statistical significance on a shoestring budget

#11

Post by AlexeyMK »

grega5 wrote: post #5

First, you really should move away from frequentist statistical testing and use Bayesian statistics instead. It is perfect for such occasions where you want to adjust your beliefs in what UX is best based on empirical data to support your decision. With collecting data you are increasing confidence in your decision rather than trying to meet an arbitrary criterion of a specific p-value. Second, the “run-in-parallel”…

Thanks for factorial design! I'll update the post to the proper nomenclature.

The frequentist/bayesian debate is not one I understand well enough to opine - do you have any reading you'd recommend for this topic?

Re: Statistical significance on a shoestring budget

#12

Post by AlexeyMK »

yorwba wrote: post #10
AlexeyMK wrote: post #9

Earlier quoted context omitted.

> Nooo! First, if one actually works, you’ve massively increased the “noise” for the other experiments I get that a bunch at some of my clients. It's a common misconception. Let's say experiment B is 10% better than control but we're also running experiment C at the same time. Since C's participants are evenly distributed across B's branches, by default they should have no impact on the other experiment. If you do a…

The more experiments you run in parallel, the more likely it becomes that at least one experiment's branches do not have an even distribution across all branches of all (combinations of) other experiments. And the more experiments you run, whether in parallel or sequentially, the more likely you're to get at least one false positive, i.e. p-hacking. XKCD is using "find a subgroup that happens to be positive" to make…

Thanks, that is a well reasoned argument!

My take is for small n (say 5 experiments at once) with lots of subjects (>10k participants per branch) and a decent hashing algorithm, the risk of uneven bucketing remains negligible. Is my intuition off?

False positives for experiments is definitely something to keep an eye on. The question to ask is what is our comfort level for trading-off between false positives and velocity. This feels similar to the IRB debate to me, where being too restrictive hurts progress more than it prevents harm.

Re: Statistical significance on a shoestring budget

#13

Post by scottfr »

grega5 wrote: post #5

First, you really should move away from frequentist statistical testing and use Bayesian statistics instead. It is perfect for such occasions where you want to adjust your beliefs in what UX is best based on empirical data to support your decision. With collecting data you are increasing confidence in your decision rather than trying to meet an arbitrary criterion of a specific p-value. Second, the “run-in-parallel”…

Disagree on using Bayesian statistics. Frequentist statistics are perfect for A/B testing.

There are so many strong biases people have about different parts about UI/UX. One of the significant benefits of A/B testing is that it lets you move ahead as a team and make decisions even when there are strongly differing opinions on your team. In these cases you can just "A/B test" and let the data decide.

But if you are using Bayesian approaches you'll transition those internal arguments to what the prior should be and it will be harder to get alignment based on the data.

Re: Statistical significance on a shoestring budget

#14

Post by lukego »

yorwba wrote: post #10
AlexeyMK wrote: post #9

Earlier quoted context omitted.

> Nooo! First, if one actually works, you’ve massively increased the “noise” for the other experiments I get that a bunch at some of my clients. It's a common misconception. Let's say experiment B is 10% better than control but we're also running experiment C at the same time. Since C's participants are evenly distributed across B's branches, by default they should have no impact on the other experiment. If you do a…

The more experiments you run in parallel, the more likely it becomes that at least one experiment's branches do not have an even distribution across all branches of all (combinations of) other experiments. And the more experiments you run, whether in parallel or sequentially, the more likely you're to get at least one false positive, i.e. p-hacking. XKCD is using "find a subgroup that happens to be positive" to make…

... and one such correction is the (simple, conservative, underused) Bonferroni Correction.

Re: Statistical significance on a shoestring budget

#15

Post by eru »

scottfr wrote: post #13
grega5 wrote: post #5

First, you really should move away from frequentist statistical testing and use Bayesian statistics instead. It is perfect for such occasions where you want to adjust your beliefs in what UX is best based on empirical data to support your decision. With collecting data you are increasing confidence in your decision rather than trying to meet an arbitrary criterion of a specific p-value. Second, the “run-in-parallel”…

Disagree on using Bayesian statistics. Frequentist statistics are perfect for A/B testing. There are so many strong biases people have about different parts about UI/UX. One of the significant benefits of A/B testing is that it lets you move ahead as a team and make decisions even when there are strongly differing opinions on your team. In these cases you can just "A/B test" and let the data decide. But if you are us…

Not necessarily.

You can present your Bayesian approaches in such a way that it's almost independent of the prior. Your output will be 'this experiment should shift your odds-ratio by so-and-so-many logits in this or that direction' instead of an absolute probability.

Re: Statistical significance on a shoestring budget

#16

Post by AlexeyMK »

lukego wrote: post #14
yorwba wrote: post #10

Earlier quoted context omitted.

The more experiments you run in parallel, the more likely it becomes that at least one experiment's branches do not have an even distribution across all branches of all (combinations of) other experiments. And the more experiments you run, whether in parallel or sequentially, the more likely you're to get at least one false positive, i.e. p-hacking. XKCD is using "find a subgroup that happens to be positive" to make…

... and one such correction is the (simple, conservative, underused) Bonferroni Correction.

Super helpful - looked it up, will aim to apply next time!

Curious how the bonferroni correction applies in cases where the overlap is partial - IE, experiment A ran from Day 1 to 14, and experiment B ran (on the same group) from days 8 to 21. Do you just apply the correction as if there was full overlap?

Re: Statistical significance on a shoestring budget

#17

Post by lukego »

AlexeyMK wrote: post #16
lukego wrote: post #14

Earlier quoted context omitted.

... and one such correction is the (simple, conservative, underused) Bonferroni Correction.

Super helpful - looked it up, will aim to apply next time! Curious how the bonferroni correction applies in cases where the overlap is partial - IE, experiment A ran from Day 1 to 14, and experiment B ran (on the same group) from days 8 to 21. Do you just apply the correction as if there was full overlap?

I believe you would apply the correction for every comparison you make regardless of the conditions. It's a conservative default to avoid accidentally p-hacking.

There might be other more specific corrections that give you power in a specific case. I don't know about that, I went Bayesian somewhere around this point myself.

Re: Statistical significance on a shoestring budget

#18

Post by RandomLensman »

lukego wrote: post #17
AlexeyMK wrote: post #16

Earlier quoted context omitted.

Super helpful - looked it up, will aim to apply next time! Curious how the bonferroni correction applies in cases where the overlap is partial - IE, experiment A ran from Day 1 to 14, and experiment B ran (on the same group) from days 8 to 21. Do you just apply the correction as if there was full overlap?

I believe you would apply the correction for every comparison you make regardless of the conditions. It's a conservative default to avoid accidentally p-hacking. There might be other more specific corrections that give you power in a specific case. I don't know about that, I went Bayesian somewhere around this point myself.

There are a bunch of procedures under the label Family-wise Error Correction, some have issues in situations with non-independence (Bonferoni can handle any dependency structure, I think).

If there are a lot of tests/comparisons could also look at controlling for the False Discovery Rate (usually increases power at the expense of more type I errors).

Re: Statistical significance on a shoestring budget

#20

Post by RA_Fisher »

Fomite wrote: post #19

There's an argument to be made that, so long as your testing fully encompasses all visitors to your site, you aren't sampling the population, you're fully observing it, and statistical significance is irrelevant.

Sites are always getting new visitors, losing old ones and the ones they’ve observed return irregularly (or commonly, or somewhere in between). So it’s not realistic to assume a given sample of visitors is the population.
Post reply on HN