Statistical significance & other A/B test pitfalls
cennydd.co.uk
Statistical significance & other A/B test pitfalls
1–10 of 16 posts
Re: Statistical significance & other A/B test pitfalls
#2Wait, so people who do A/B tests didn’t already do that? It drives me absolutely crazy when I don‘t have any measure to assess how likely or unlikely it is for some difference to be random.
Re: Statistical significance & other A/B test pitfalls
#3Wait, so people who do A/B tests didn’t already do that? It drives me absolutely crazy when I don‘t have any measure to assess how likely or unlikely it is for some difference to be random.
I think most people do not do this because they do not know it is important, or like me, they do not understand the theory behind it. Neither did I know how to do it in practice.
Re: Statistical significance & other A/B test pitfalls
#4Wait, so people who do A/B tests didn’t already do that? It drives me absolutely crazy when I don‘t have any measure to assess how likely or unlikely it is for some difference to be random.
No, people who do A/B tests have known this for years. It is the wannabes who haven't sat down and figured out the statistics who run into trouble. See http://elem.com/~btilly/effective-ab-testing/ for an OSCON tutorial that I did on the topic a couple of years ago, which includes all the gory statistical detail you could want.
Furthermore I note with interest that 2 of the 3 statistical techniques he named (Student's t test and ANOVA) only apply to cases where the observed variables are themselves normally distributed. Which is not a good description of binary yes/no outcomes. As for the remaining test, it is appropriate to use a chi-square, but statisticians tell us that the g-test is preferable.
Re: Statistical significance & other A/B test pitfalls
#5I have a test to make.
Many people think they will get will become millionaires if they follow the style of person X.
Person X is like a trial in which a coin was tossed 10000 times and got 6000 heads.
Since there is no information about the others persons, the others trials, many choose to follow the illogical thinking that they will succeed in the same way.
Re: Statistical significance & other A/B test pitfalls
#6It's disturbing to me how p It seems to me if you have even a somewhat popular app, you're gathering enough data that you can afford to use p < 0.001 and avoid a lot of the complexities of statistical analysis that comes from p < 0.05. If you don't have enough data to reach p < 0.001, it's probably better to work more on increasing traffic than getting the piddling gains from A/B testing so early.
Re: Statistical significance & other A/B test pitfalls
#7Wait, so people who do A/B tests didn’t already do that? It drives me absolutely crazy when I don‘t have any measure to assess how likely or unlikely it is for some difference to be random.
I was quite surprised to find that the linked website designed to showcase A/B tests doesn't even hint at things like statistical significance or confidence intervals for the improvements
Re: Statistical significance & other A/B test pitfalls
#8Wait, so people who do A/B tests didn’t already do that? It drives me absolutely crazy when I don‘t have any measure to assess how likely or unlikely it is for some difference to be random.
No, people who do A/B tests have known this for years. It is the wannabes who haven't sat down and figured out the statistics who run into trouble. See http://elem.com/~btilly/effective-ab-testing/ for an OSCON tutorial that I did on the topic a couple of years ago, which includes all the gory statistical detail you could want. Furthermore I note with interest that 2 of the 3 statistical techniques he named (Student'…
I don't see the problem. The total is very nearly normally distributed by the central limit theorem, is it not?
Re: Statistical significance & other A/B test pitfalls
#9Earlier quoted context omitted.
No, people who do A/B tests have known this for years. It is the wannabes who haven't sat down and figured out the statistics who run into trouble. See http://elem.com/~btilly/effective-ab-testing/ for an OSCON tutorial that I did on the topic a couple of years ago, which includes all the gory statistical detail you could want. Furthermore I note with interest that 2 of the 3 statistical techniques he named (Student'…
I don't see the problem. The total is very nearly normally distributed by the central limit theorem, is it not?
The total is indeed nearly normally distributed, but the rate of convergence (particularly in the tails) is not fast enough to avoid having those very sensitive tests give wrong results.
Were it otherwise there would have been no need to develop the chi-square test. It would have been entirely redundant. (It actually is redundant because we have the g-test. But evaluating the chi-square test just involves taking squares, while the g-test involves taking natural logarithms. This made the less accurate chi-square test much easier to do when people didn't have computers to calculate it on. Today we should use the g-test, but few people have heard of it.)
Re: Statistical significance & other A/B test pitfalls
#10Earlier quoted context omitted.
I don't see the problem. The total is very nearly normally distributed by the central limit theorem, is it not?
The total is indeed nearly normally distributed, but the rate of convergence (particularly in the tails) is not fast enough to avoid having those very sensitive tests give wrong results. Were it otherwise there would have been no need to develop the chi-square test. It would have been entirely redundant. (It actually is redundant because we have the g-test. But evaluating the chi-square test just involves taking squa…
Ah, right. I spent a while drawing up a proper plot of the likelihood of the difference and the normal approximation of the difference, and saw that the normal had too small a variance. The effect is still pretty credible in the OP, though.