Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

e10v.me

Post reply on HN
Print view View original

1–10 of 52 posts

Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#1

Post by e10v_me »

I'm excited to introduce tea-tasting, a Python package for the statistical analysis of A/B tests

It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.

tea-tasting supports a wide range of data backends, including BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and more, all thanks to Ibis.

I consider it ready for important tasks and use it for the analysis of switchback experiments in my work.

Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests
e10v.me

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#5

Post by deugtniet »

I guess I'm not very versed in website A/B testing, but wouldn't it be much better to analyze these results in a regression framework where you can correct for the covariates?

On top of this, logistic regression makes your units a lot more interpretable than just looking at differences in means. I.E. The odds of buying something are 1.1 when you are assigned in group B.

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#6

Post by e10v_me »

prashp wrote: post #3

It's called tea-tasting but doesn't include Fisher's exact test :( https://en.wikipedia.org/wiki/Fisher%27s_exact_test?useskin=...

Yeah, I know :) But it's in the roadmap. Btw, aren't Barnard's test or Boschloo's test better alternatives?

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#7

Post by thegginthesky »

Great package! I'll test it out on my free time.

I'm wondering if you'd like to accept a contribution for Bayesian AB Testing, based on this whitepaper[0] and developed in Numpy.

If so, we can chat at my email gbenatt92 at zohomail dot com, or I can open a draft PR to discuss the code and paper.

[0]https://vwo.com/downloads/VWO_SmartStats_technical_whitepape...

Re: Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests

#10

Post by teej »

0cf8612b2e1e wrote: post #9

What is a good resource for someone looking to learn more about A/B testing? Not specifically about website dark pattern optimization, but fine if that is the framing device.

I recommend “Trustworthy Online Controlled Experiments”. If you’re only going to read one book about it, it should be this one. It will walk you through why we experiment, how it’s typically done, and how to use them to improve your decision making.
Post reply on HN