These look great. For the sake of the permalinks, I'd love to see these hosted on another domain. Maybe a Wizard app domain, to increase brand awareness?
A github page would also be a good idea
Announcing Evan's Awesome A/B Tools
21–26 of 26 posts
Re: Announcing Evan's Awesome A/B Tools
#22As a math/stats/data person who doesn't dabble much in web optimization -- can someone explain to me what's awesome about this? Not to belittle this nice package -- it looks like a basic stats calculator for calculating sample size confidence levels with friendly visualization, and I'm just trying to understand what's being valued on the market/industry right now. Is it because current A/B testing software doesn't pr…
Like all of A/B testing, it's applying a _very_ old statistical method (Chi-Square was one of the first modern statistical techniques--by that I mean it's 113 years old) to an area where statistics has not commonly been used. This makes it seem wonderful and novel as countless people suddenly realize that statistics can be applied to fields that were previously untouched by quantitative analyses.
The statistics being used in the A/B testing world is stuff you would've learned in your very first statistics class, it seems. From the success of Optimizely and VWO that the focus is definitely more on the viz and presentation than it is on using any cutting-edge techniques.
Re: Announcing Evan's Awesome A/B Tools
#23"Use a dedicated statistical package from the '80s"
Is your Wizard app not also a dedicated statistical package? Also, I'm being pedantic here, but how many "dedicated statistical packages" are actually from the 80s? The only ones that come to mind are Stata and Statistica.
Is there a way to view the source code or formulas you use on your pages? There's been a strong push in the academic statistics world for reproducible research, which means public data, open source statistical code.
I ask because I'm curious about your two-sample t-test. Does it pool the variances for all values of the two standard deviations? Pooling doesn't make sense when one sd is 50 and the other is 2...
Re: Announcing Evan's Awesome A/B Tools
#24As a math/stats/data person who doesn't dabble much in web optimization -- can someone explain to me what's awesome about this? Not to belittle this nice package -- it looks like a basic stats calculator for calculating sample size confidence levels with friendly visualization, and I'm just trying to understand what's being valued on the market/industry right now. Is it because current A/B testing software doesn't pr…
Like all of A/B testing, it's applying a _very_ old statistical method (Chi-Square was one of the first modern statistical techniques--by that I mean it's 113 years old) to an area where statistics has not commonly been used. This makes it seem wonderful and novel as countless people suddenly realize that statistics can be applied to fields that were previously untouched by quantitative analyses. The statistics being…
Gotcha, and thanks -- it just seemed trivial and I was under the (false) assumption that confidence levels and selecting appropriate sample size should be common knowledge, given how much polls are used in day-to-day life.
Good to know there's plenty of opportunity to bring better stats to high tech. Of course, I understand a lot of the value comes from making those things applicable and meaningful to the users...
Re: Announcing Evan's Awesome A/B Tools
#25Are there any usefull resources for learning the statistic methods needed for A/B test evaluation for beginners?
Re: Announcing Evan's Awesome A/B Tools
#26Earlier quoted context omitted.
Like all of A/B testing, it's applying a _very_ old statistical method (Chi-Square was one of the first modern statistical techniques--by that I mean it's 113 years old) to an area where statistics has not commonly been used. This makes it seem wonderful and novel as countless people suddenly realize that statistics can be applied to fields that were previously untouched by quantitative analyses. The statistics being…
Gotcha, and thanks -- it just seemed trivial and I was under the (false) assumption that confidence levels and selecting appropriate sample size should be common knowledge, given how much polls are used in day-to-day life. Good to know there's plenty of opportunity to bring better stats to high tech. Of course, I understand a lot of the value comes from making those things applicable and meaningful to the users...
Power analysis and CI's should be elementary, but I would assert that they are actually not commonplace. Most people have a very surface-level understanding of the latter, and little understanding of the former. In my opinion, A/B Testing has actually done a great service to power analysis. I have seen many experiments in the academic world (social sciences are somewhat notorious for this) forgoing the power analysis for various reasons (fear: they would not be able to get the sample size needed for 80% power, inability to control sample size: you take whatever you can get with a convenience sample). As a statistician, I breathe a sigh of relief with the amount of emphasis power analysis receives in the A/B world. It's a step in the right direction (if you're an acolyte to the dark world of Neyman-Pearson).
As for bringing better stats to high tech, I've thought of this as a wonderful challenge. I'd especially like to see more focus on not violating modeling assumptions (more non and semi-parametrics), and using some more modern techniques from the ML and Bayes literature.
Hypothesis testing is so last century :). Would love to discuss it further with some similarly-inclined HN folks.
Sorry for all the parentheticals. You'd think I was a lisp programmer with the amount of parenthesis I used.