Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical significance & other A/B test pitfalls

cennydd.co.uk

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Statistical significance & other A/B test pitfalls

#11

Post by btilly »

shalmanese wrote: post #6

It's disturbing to me how p It seems to me if you have even a somewhat popular app, you're gathering enough data that you can afford to use p < 0.001 and avoid a lot of the complexities of statistical analysis that comes from p < 0.05. If you don't have enough data to reach p < 0.001, it's probably better to work more on increasing traffic than getting the piddling gains from A/B testing so early.

People blindly stopping at 0.05 is doubly worrying given that people tend to stop an A/B test as soon as it shows significance. That gives them multiple chances to be wrong. Furthermore if you are getting close to significance very fast, then strong significance is close behind, so why not wait?

That said, if a test has been running for a while and you don't have an answer, it can run for a looong time before it finishes. In my A/B testing tutorial I explored that starting at http://www.elem.com/~btilly/effective-ab-testing/#slide59 (just use the arrow keys to move forwards and backwards through those slides). I found that depending on whether random fluctuations took you in the same direction as the underlying bias or the opposite, there tends to be an order of magnitude difference in how long the test takes to run. Furthermore whichever is leading after many observations is usually really better, and in the worst case is overwhelmingly likely to not be much worse. Therefore there are times when it really is better to declare an answer and move on.

If you wish to formalize this, you could use the strategy used by some medical trials where they decide in advance what confidence levels will cause them to cut off early after 100 trials, 1000, trials, 10,000 trials, or to go to (say) 50,000 trials. And then they arrange that the sum of the odds that they make an early mistake are below some acceptable threshold.

Re: Statistical significance & other A/B test pitfalls

#12

Post by sesqu »

sesqu wrote: post #10
btilly wrote: post #9

Earlier quoted context omitted.

The total is indeed nearly normally distributed, but the rate of convergence (particularly in the tails) is not fast enough to avoid having those very sensitive tests give wrong results. Were it otherwise there would have been no need to develop the chi-square test. It would have been entirely redundant. (It actually is redundant because we have the g-test. But evaluating the chi-square test just involves taking squa…

Ah, right. I spent a while drawing up a proper plot of the likelihood of the difference and the normal approximation of the difference, and saw that the normal had too small a variance. The effect is still pretty credible in the OP, though.

Noprocrast caught out my attempt to edit. The normal variance is too large.

Here's the plot. Black for discretized(n=1000) binomial likelihood, red for normal approximation. The effect is clear, but a t-test won't show it. I'm not familiar with the theory behind the g-test, but there's clearly a lot of room for improvement at these sample sizes.

http://img693.imageshack.us/img693/4880/bindiff.png

Re: Statistical significance & other A/B test pitfalls

#13

Post by btilly »

sesqu wrote: post #12
sesqu wrote: post #10

Earlier quoted context omitted.

Ah, right. I spent a while drawing up a proper plot of the likelihood of the difference and the normal approximation of the difference, and saw that the normal had too small a variance. The effect is still pretty credible in the OP, though.

Noprocrast caught out my attempt to edit. The normal variance is too large . Here's the plot. Black for discretized(n=1000) binomial likelihood, red for normal approximation. The effect is clear, but a t-test won't show it. I'm not familiar with the theory behind the g-test, but there's clearly a lot of room for improvement at these sample sizes. http://img693.imageshack.us/img693/4880/bindiff.png

It looks like you forgot to rescale the binomial distribution.

If X_i is a series of independent, identically distributed random variables with mean m and variance v, then X_1 + X_2 + ... + X_n is approximately a normal variable with mean nm and variance vn. Therefore

(X_1 + X_2 + ... + X_n - nm)/sqrt(vn)

is approximately a standard normal.

If you draw that graph, visually the two lines should lie right on top of each other. To see the problem you need to zoom in on the tail and blow it up, and only then will you see the issues with the convergence.

Re: Statistical significance & other A/B test pitfalls

#14

Post by _delirium »

Another common issue he doesn't mention: using observed differences (or observed significance-test values) as the stopping criterion. The common statistical-significance tests don't work if the decision when to stop collecting data is dependent on the observed levels of significance. Instead, you must ahead of time decide how many trials to do, and stick to that decision, or use more complicated significance tests. (This is the "multiple testing" problem.)

For example, it works to flip two coins 50 times each, and then run a statistical-significance test. It does not work to flip two coins 50 times each, run a test; if no significance yet, continue to 100, then 150, etc. until you either find a significant difference or give up. That greatly increases the chance that you'll get a spurious significance, because your stopping is biased in favor of answering "yes": if you found a difference at 50, you don't go on to 100 (where maybe the difference would disappear again), but if you didn't find a difference at 50, you do go on to 100.

Put differently, it's using separate p-values for "what is the chance I could've gotten this result in [50|100|150|...] trials with unweighted coins?" to reject the null hypothesis each time, as if they were independent, but the null hypothesis for the entire series has to be the union, "what is the chance I could've seen this result at any of the 50, 100, 150, or 200, ... stopping points with unweighted coins?", which is higher. Yet that's exactly how many A/B tests are done: you start collecting data, and let the trials run until you find "significant" differences or give up.

(It's possible to set up a series of tests where you choose when to stop based on observed values, but you have to use different statistical machinery than the common significance-tests.)

Re: Statistical significance & other A/B test pitfalls

#15

Post by sesqu »

btilly wrote: post #13
sesqu wrote: post #12

Earlier quoted context omitted.

Noprocrast caught out my attempt to edit. The normal variance is too large . Here's the plot. Black for discretized(n=1000) binomial likelihood, red for normal approximation. The effect is clear, but a t-test won't show it. I'm not familiar with the theory behind the g-test, but there's clearly a lot of room for improvement at these sample sizes. http://img693.imageshack.us/img693/4880/bindiff.png

It looks like you forgot to rescale the binomial distribution. If X_i is a series of independent, identically distributed random variables with mean m and variance v, then X_1 + X_2 + ... + X_n is approximately a normal variable with mean nm and variance vn. Therefore (X_1 + X_2 + ... + X_n - nm)/sqrt(vn) is approximately a standard normal. If you draw that graph, visually the two lines should lie right on top of eac…

You're right, I did have the wrong scale, and should have realized. I believe this plot to be correct. Black is binomial, red is normal, both for the difference in conversion rates.

http://img267.imageshack.us/img267/6691/binnormdifflikelihoo...

The normal approximation seems reasonable, and so a t-test shouldn't cause problems. I made a plot of the relative error of the normal approximation; the tails are indeed too fat. In particular, there's a bump to the left of 0, so a t-test would slightly overestimate p.

http://img205.imageshack.us/img205/6691/binnormdifflikelihoo...

Re: Statistical significance & other A/B test pitfalls

#16

Post by btilly »

sesqu wrote: post #15
btilly wrote: post #13

Earlier quoted context omitted.

It looks like you forgot to rescale the binomial distribution. If X_i is a series of independent, identically distributed random variables with mean m and variance v, then X_1 + X_2 + ... + X_n is approximately a normal variable with mean nm and variance vn. Therefore (X_1 + X_2 + ... + X_n - nm)/sqrt(vn) is approximately a standard normal. If you draw that graph, visually the two lines should lie right on top of eac…

You're right, I did have the wrong scale, and should have realized. I believe this plot to be correct. Black is binomial, red is normal, both for the difference in conversion rates. http://img267.imageshack.us/img267/6691/binnormdifflikelihoo... The normal approximation seems reasonable, and so a t-test shouldn't cause problems. I made a plot of the relative error of the normal approximation; the tails are indeed too…

The problem is that when you're near confidence you're in the tail, and the student t-test is extremely sensitive to the shape of said tail. If n is large enough this difference will be washed away as everything converges to normals. But with smaller sample sizes the difference can be quite significant.
Post reply on HN