Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

e10v.me

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#21

Post by asdasdsddd »

welder wrote: post #18
asdasdsddd wrote: post #14

Imo, with a/b tests, its really easy to get sucked into the 30 different analysis algos, but the most important thing by far is experiment hygiene

And knowing beforehand when you won't get enough exposures to reach significance. Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years. I don't use CUPED in my tests... how much can it reduce wait times?

I don't think CUPED is super useful if you just stratify your users properly before the experiment begins.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#23

Post by welder »

crystal_revenge wrote: post #20
welder wrote: post #18

Earlier quoted context omitted.

And knowing beforehand when you won't get enough exposures to reach significance. Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years. I don't use CUPED in my tests... how much can it reduce wait times?

Strictly speaking you don't need to wait for some arbitrary significance threshold. I don't know why so many people treat website A/B tests as similar to carefully, traditional nhst controlled experiments. Website A/B testing is much better thought of as an optimization problem rather than a true hypothesis test. What's really important if you want to improve a website via A/B testing is a constant stream of new hypo…

> You can call tests "early"

Another way to say that is: you can randomly pick a winner

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#24

Post by e10v_me »

deugtniet wrote: post #5

I guess I'm not very versed in website A/B testing, but wouldn't it be much better to analyze these results in a regression framework where you can correct for the covariates? On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.

Thank you for the interest and for the suggestion.

Yes, one can analyze A/B tests in a regression framework. In fact, CUPED is an equivalent to the linear regression with a single covariate.

Would it be better? It depends on the definition of "better". There are several factors to consider. Scientific rigor is one of them. So is the computational efficiency.

A/B tests are usually conducted at scale of thousands of randomization units (actually it's more like tens or hundreds of thousands). There are two consequences:

1. Computational efficiency is very important, especially if we take into account the number of experiments and the number of metrics. And pulling granular data into a Python environment and fitting a regression is much less efficient than calculating aggregated statistics like mean and variance.

2. I didn't check, but I'm pretty sure that, at such scale, logistic and linear regressions' results will be very close, if not equal.

And even if, for some reason, there is a real need to analyze a test using logistic model, multi-level model, or a clustered error, in tea-tasting, it's possible via custom metrics: https://tea-tasting.e10v.me/custom-metrics/

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#25

Post by crystal_revenge »

deugtniet wrote: post #5

I guess I'm not very versed in website A/B testing, but wouldn't it be much better to analyze these results in a regression framework where you can correct for the covariates? On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.

This is the correct approach, but having done A/B testing for many years (and basically moved away from this area of work), nobody in the industry really cares about understanding the problem they care about prompting themselves as experts and creating the illusion of rigorous marketting.

Correct A/B testing should involved starting with an A/A test to validate the setup, building a basic causal model of what you expect the treatment impact to be, controlling of covariates, and finally ensuring that when the causal factor is controlled for the results change as expected.

But even the "experts" I've read in this area largely focus on statistical details that honestly don't matter (and if they do the change you're proposing is so small that you shouldn't be wasting time on it).

In practice if you need "statistical significance" to determine if change has made an impact on your users you're already focused on problems that are too small to be worth your time.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#26

Post by e10v_me »

asdasdsddd wrote: post #14

Imo, with a/b tests, its really easy to get sucked into the 30 different analysis algos, but the most important thing by far is experiment hygiene

I cannot agree more. It's one of the reasons I've developed the package. With tea-tasting, I can save time and focus on the more important tasks, like experiment design.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#27

Post by crystal_revenge »

e10v_me wrote: post #24
deugtniet wrote: post #5

I guess I'm not very versed in website A/B testing, but wouldn't it be much better to analyze these results in a regression framework where you can correct for the covariates? On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.

Thank you for the interest and for the suggestion. Yes, one can analyze A/B tests in a regression framework. In fact, CUPED is an equivalent to the linear regression with a single covariate. Would it be better? It depends on the definition of "better". There are several factors to consider. Scientific rigor is one of them. So is the computational efficiency. A/B tests are usually conducted at scale of thousands of ra…

> And pulling granular data into a Python environment and fitting a regression is much less efficient than calculating aggregated statistics like mean and variance.

This is not true. You almost never need to perform logistic regression on individual observations. Consider that estimating a single Bernoulli rv on N observations is the same as estimate a single Binomial rv for k/N. Most common statistical software (e.g. statsmodels) will support this grouped format.

If all of our covariates a discrete categories (which is typically the case for A/B tests) then you only need to regression on the number of examples equal to the number of unique configurations of the variables.

That is if you're running an A/B test on 10 million users across 50 states and 2 variants you only need 100 observations for your final model.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#28

Post by gatopingado »

cschmidt wrote: post #11

It would probably be good to have something considering multiple comparisons (False Discovery Rate, Bonferroni correction), which is often the bane of running a whole series of A/B tests. And, as another poster has mentioned, an anytime approach that is resistant to early stopping due to peaking [1]. For those who haven't read about Fisher's tea experiment: There was a woman who claimed she could tell if the milk was…

For anyone interested in anytime-valid testing, I wrote a Python library [1] implementing multinomial and time inhomogeneous Bernoulli / Poisson process tests based in [2].

[1] https://github.com/assuncaolfi/savvi/

[2] https://openreview.net/forum?id=a4zg0jiuVi

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#29

Post by crystal_revenge »

welder wrote: post #23
crystal_revenge wrote: post #20

Earlier quoted context omitted.

Strictly speaking you don't need to wait for some arbitrary significance threshold. I don't know why so many people treat website A/B tests as similar to carefully, traditional nhst controlled experiments. Website A/B testing is much better thought of as an optimization problem rather than a true hypothesis test. What's really important if you want to improve a website via A/B testing is a constant stream of new hypo…

> You can call tests "early" Another way to say that is: you can randomly pick a winner

Of course at the extreme you are over tuning for exploitation but in practice it's never completely random. You always have some information about the probably winner, so long as the P(A>B|obs) is not 0.5

Taking a long time to reach "significance" just means there is a small difference between the two variants, so it's better to just choose one and the try the next challenger which might have a larger difference.

In the early stages of running A/B tests being 90% certain that one variant is superior is perfectly fine so long as you have another challenger ready. Conversely, In the later stages of a mature website when you're searching for minor gains you probably want a much higher level of certainty that then standard 95%.

In either case thinking in terms of arbitrary significance thresholds doesn't make that much sense for A/B testing.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#30

Post by e10v_me »

NightMKoder wrote: post #8

Would be great if it included sequential sampling as well: https://www.evanmiller.org/ab-testing/sequential.html . Especially given how A/B tests usually get run in product companies, a peek proof method helps quite a bit.

I will consider it. Thank you for the suggestion.
Post reply on HN