31–40 of 52 posts
I won't use any library that depends on numpy because of all the install issues in the past. Can't you do these tests with pure Python these days?
Python packaging is a mess, but compared to issues with Torch or Nvidia stuff, Numpy has been a cakewalk whether using pip, conda, poetry, rye, etc.
Earlier quoted context omitted.
And knowing beforehand when you won't get enough exposures to reach significance. Not many people have enough traffic to A/B test small effects and reach significance without running the test for multiple years. I don't use CUPED in my tests... how much can it reduce wait times?
I don't think CUPED is super useful if you just stratify your users properly before the experiment begins.
Earlier quoted context omitted.
Thank you for the interest and for the suggestion. Yes, one can analyze A/B tests in a regression framework. In fact, CUPED is an equivalent to the linear regression with a single covariate. Would it be better? It depends on the definition of "better". There are several factors to consider. Scientific rigor is one of them. So is the computational efficiency. A/B tests are usually conducted at scale of thousands of ra…
> And pulling granular data into a Python environment and fitting a regression is much less efficient than calculating aggregated statistics like mean and variance. This is not true. You almost never need to perform logistic regression on individual observations. Consider that estimating a single Bernoulli rv on N observations is the same as estimate a single Binomial rv for k/N. Most common statistical software (e.g…
Interesting, I didn't know this about statsmodels. But maybe documentation a bit misleading: "A nobs x k array where nobs is the number of observations and k is the number of regressors". Source: https://www.statsmodels.org/stable/generated/statsmodels.gen...
I would be grateful for the references on how to apply statsmodels for solving logistic model using only aggregated statistics. Or not statsmodels. Any references will do.
I guess I'm not very versed in website A/B testing, but wouldn't it be much better to analyze these results in a regression framework where you can correct for the covariates? On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.
This is the correct approach, but having done A/B testing for many years (and basically moved away from this area of work), nobody in the industry really cares about understanding the problem they care about prompting themselves as experts and creating the illusion of rigorous marketting. Correct A/B testing should involved starting with an A/A test to validate the setup, building a basic causal model of what you exp…
I think the dumb underlying question I have is - how does one do experimental design
Edit: and if you aren’t seeing giant obvious improvements, try improving something else (I get the idea that my B is going to be so obvious that there is no need to worry about stats - if it’s not that’s a signal to chnage something else?
Congrats! Let's say I have 5 variants I want to try. Does this package have anything to help with realtime experiment design, where I stop trying the less-promising variants, and focus my experimental budget on the more promising variants?
Very cool! Just curious, would you consider adding more exotic experimental design setups like Latin Square Design to the roadmap?
Earlier quoted context omitted.
> And pulling granular data into a Python environment and fitting a regression is much less efficient than calculating aggregated statistics like mean and variance. This is not true. You almost never need to perform logistic regression on individual observations. Consider that estimating a single Bernoulli rv on N observations is the same as estimate a single Binomial rv for k/N. Most common statistical software (e.g…
> Most common statistical software (e.g. statsmodels) will support this grouped format. Interesting, I didn't know this about statsmodels. But maybe documentation a bit misleading: "A nobs x k array where nobs is the number of observations and k is the number of regressors". Source: https://www.statsmodels.org/stable/generated/statsmodels.gen... I would be grateful for the references on how to apply statsmodels for s…
So that will be a bit different than r style formula's using cbind, but yes if you only have a few categories of data using weights makes sense. (Even many of sklearn's functions allow you to pass in weights.)
I have not worked out closed form for logit regression, but for Poisson regression you can get closed form for the incident rate ratio, https://andrewpwheeler.com/2024/03/18/poisson-designs-and-mi.... So no need to use maximum likelihood at all in that scenario.
Great package! I'll test it out on my free time. I'm wondering if you'd like to accept a contribution for Bayesian AB Testing, based on this whitepaper[0] and developed in Numpy. If so, we can chat at my email gbenatt92 at zohomail dot com, or I can open a draft PR to discuss the code and paper. [0] https://vwo.com/downloads/VWO_SmartStats_technical_whitepape...
Regarding your question, first, I'd like to understand what problem you want to solve, and whether this approach will be useful for other users of tea-tasting.
It would probably be good to have something considering multiple comparisons (False Discovery Rate, Bonferroni correction), which is often the bane of running a whole series of A/B tests. And, as another poster has mentioned, an anytime approach that is resistant to early stopping due to peaking [1]. For those who haven't read about Fisher's tea experiment: There was a woman who claimed she could tell if the milk was…
1. Experiments with 3 or more variants are quite rare in my practice. I usually try to avoid them.
2. In my opinion, the Bonferroni correction is just wrong. It's too pessimistic. There are better methods though.
3. The choice of alpha is subjective. Why use a precise smart method to adjust a subjective parameter? Just choose another subjective alpha, a smaller one :)
But I can change my opinion if I see a good argument.
I won't use any library that depends on numpy because of all the install issues in the past. Can't you do these tests with pure Python these days?