A/B Testing, from scratch
alfredo.motta.name
A/B Testing, from scratch
1–10 of 15 posts
Re: A/B Testing, from scratch
#2Glad to read this. So much so-called A/B testing done is of the naive variety he outlines.
Much more here: http://www.exp-platform.com/Pages/default.aspx
disclosure: I worked with ExP a while ago.
Re: A/B Testing, from scratch
#3Glad to read this. So much so-called A/B testing done is of the naive variety he outlines. Much more here: http://www.exp-platform.com/Pages/default.aspx disclosure: I worked with ExP a while ago.
Second the above. The team at MS have done incredible work around the pragmatics of being able to trust the findings of the online experiment and possible areas of confounds. Anything Kohavi touches is worth paying attention to.
Re: A/B Testing, from scratch
#4A lot of math detail, but a naive reader reading it will still make the mistake outlined in http://www.evanmiller.org/how-not-to-run-an-ab-test.html. And a less naive reader will still be left with no guidance on how to make real-world decisions if you weren't so lucky as to get a strong statistical result.
Here is a much, much simpler strategy that I can suggest for business users.
1. Figure out the longest that you can afford to run your test for.
2. Figure out the number of conversions N you expect in that time.
3. Start running your test, randomly splitting visitors into A and B.
4. Stop the test as soon as one version has sqrt(N) more conversions than the other. Else wait until you get to N conversions between them, and go with what is ahead.
Here are some comments on the procedure.
Stopping before N/2 total conversions is roughly the same certainty as stopping at 95% confidence. Stopping after that is an admission that you are crossing your fingers and going with an educated guess, and it is not feasible to collect enough data to get a better answer.
If one version has a conversion rate which is (1 + 2/sqrt(n)) better than the other, you pretty reliably choose right. The flip side of this is that you'll make a lot of mistakes if you get much below that threshold. If those potential errors are too big, then A/B testing is not going to work well for you because you don't have enough data for statistics.
Re: A/B Testing, from scratch
#5There's a possible big language barrier here from "urn". I guessed it was an acronym, wikipedia[1] seemed like no help until I realized that "urn" if for an actual vessel-like object (called an urn) so I could find the correct article[2].
Which turns out to be such a simple concept, but maybe a note could help future international readers.
[1] https://en.wikipedia.org/wiki/Urn_%28disambiguation%29 [2] https://en.wikipedia.org/wiki/Urn_problem
Re: A/B Testing, from scratch
#6A lot of math detail, but a naive reader reading it will still make the mistake outlined in http://www.evanmiller.org/how-not-to-run-an-ab-test.html . And a less naive reader will still be left with no guidance on how to make real-world decisions if you weren't so lucky as to get a strong statistical result. Here is a much, much simpler strategy that I can suggest for business users. 1. Figure out the longest that yo…
btilly where does this approach come from?
Re: A/B Testing, from scratch
#7A lot of math detail, but a naive reader reading it will still make the mistake outlined in http://www.evanmiller.org/how-not-to-run-an-ab-test.html . And a less naive reader will still be left with no guidance on how to make real-world decisions if you weren't so lucky as to get a strong statistical result. Here is a much, much simpler strategy that I can suggest for business users. 1. Figure out the longest that yo…
btilly where does this approach come from?
My scratchbook. :-)
I keep meaning to write it up.
Mathematically it is similar http://www.evanmiller.org/sequential-ab-testing.html but with the difference that Evan produced a stopping rule that is a 95% confidence interval, while mine is stopping at 95% confidence that you won't come to a different conclusion by the end of the test. Otherwise the idea of the analysis is the same.
(Note, under the null hypothesis reaching sqrt(N) then crossing 0 is exactly as hard as reading 2 sqrt(N) as can be seen by taking any possible sequence that does one, and swapping the values after first reaching sqrt(N) to get a sequence that does the other.)
Re: A/B Testing, from scratch
#8Good read, but pretty dense (or perhaps I'm just dense).
Personally, I get more practical value from http://elem.com/~btilly/effective-ab-testing/.
Re: A/B Testing, from scratch
#9This article feels similar to "how to draw an owl".
Re: A/B Testing, from scratch
#10>"In order to estimate what is the true mean of our variants statisticians rely on the Central Limit Theorem (CLT) 6 which states that the sampling distribution of any statistic will be normal or nearly normal, if the sample size is large enough."
Wow, no. This is a very dangerous misconception. First sentence of the wikipedia page gives a much better definition:
"In probability theory, the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed, regardless of the underlying distribution."