Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

Post by [deleted] » Wed, Jul 31, 2024, 10:29 AM UTC

cschmidt wrote: post #49

e10v_me wrote: post #39

Earlier quoted context omitted.

I thought about multiple comparison corrections. Here what my thoughts were: 1. Experiments with 3 or more variants are quite rare in my practice. I usually try to avoid them. 2. In my opinion, the Bonferroni correction is just wrong. It's too pessimistic. There are better methods though. 3. The choice of alpha is subjective. Why use a precise smart method to adjust a subjective parameter? Just choose another subject…

If you work for a large website (as I used to), they probably run hundreds of tests a week across various groups. So false positives are a real problem, and often you don't see the gain suggested by the A/B when rolling it out. I agree that Bonferroni is often too pessimistic. If you Bonferroni correct you'll usually find nothing is significant. And I take your point that you could adjust the $\alpha$. But then of co…