Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

e10v.me

Post reply on HN
Print view View original

11–20 of 52 posts

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#11

Post by cschmidt »

It would probably be good to have something considering multiple comparisons (False Discovery Rate, Bonferroni correction), which is often the bane of running a whole series of A/B tests. And, as another poster has mentioned, an anytime approach that is resistant to early stopping due to peaking [1].

For those who haven't read about Fisher's tea experiment: There was a woman who claimed she could tell if the milk was put into the cup before or after pouring the tea. Fished didn't think so, and developed the experimental technique to test this idea. Indeed she could, getting them all right iirc.

[1] see https://media.trustradius.com/product-downloadables/UP/GB/AD... for a discussion of the problems with a t-test. There is also a more detailed whitepaper from Optimizely somewhere

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#12

Post by crimsoneer »

0cf8612b2e1e wrote: post #9

What is a good resource for someone looking to learn more about A/B testing? Not specifically about website dark pattern optimization, but fine if that is the framing device.

Goes way beyond t-tests, but I really like this free online book on causal inference more broadly

https://matheusfacure.github.io/python-causality-handbook/la...

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#16

Post by e10v_me »

teej wrote: post #10
0cf8612b2e1e wrote: post #9

What is a good resource for someone looking to learn more about A/B testing? Not specifically about website dark pattern optimization, but fine if that is the framing device.

I recommend “Trustworthy Online Controlled Experiments”. If you’re only going to read one book about it, it should be this one. It will walk you through why we experiment, how it’s typically done, and how to use them to improve your decision making.

Agree. I also suggest looking at Alex Deng's unfinished book on causal inference and, particularly, A/B testing: https://alexdeng.github.io/causal/

Alex Deng worked with Ron Kohavi at Microsoft Analysis and Experimentation Team and co-autored many important papers on the topic, including paper about CUPED.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#18

Post by welder »

asdasdsddd wrote: post #14

Imo, with a/b tests, its really easy to get sucked into the 30 different analysis algos, but the most important thing by far is experiment hygiene

And knowing beforehand when you won't get enough exposures to reach significance.

Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years.

I don't use CUPED in my tests... how much can it reduce wait times?

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#19

Post by welder »

cschmidt wrote: post #11

It would probably be good to have something considering multiple comparisons (False Discovery Rate, Bonferroni correction), which is often the bane of running a whole series of A/B tests. And, as another poster has mentioned, an anytime approach that is resistant to early stopping due to peaking [1]. For those who haven't read about Fisher's tea experiment: There was a woman who claimed she could tell if the milk was…

And the Student's t-test which was named so because William Sealy Gosset's employer (Guinness beer) allowed him to publish it anonymously, so he published using the pseudonym "Student".

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#20

Post by crystal_revenge »

welder wrote: post #18
asdasdsddd wrote: post #14

Imo, with a/b tests, its really easy to get sucked into the 30 different analysis algos, but the most important thing by far is experiment hygiene

And knowing beforehand when you won't get enough exposures to reach significance. Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years. I don't use CUPED in my tests... how much can it reduce wait times?

Strictly speaking you don't need to wait for some arbitrary significance threshold. I don't know why so many people treat website A/B tests as similar to carefully, traditional nhst controlled experiments. Website A/B testing is much better thought of as an optimization problem rather than a true hypothesis test.

What's really important if you want to improve a website via A/B testing is a constant stream of new hypotheses (i.e. new variants). You can call tests "early" so long as you have new tests lined up it boils don't to a classic exploitation/exploration problem. In fact, in early development rapid iteration often yields superior results to waiting for significance.

As a website matures and reaches closer to some theoretical optimal conversion point, then it starts becoming increasing important to wait until you are very certain of an improvement. But if you're just starting A/B testing, more iteration will yield greater success than more certainty.

Post reply on HN