Announcing Evan's Awesome A/B Tools
evanmiller.org
Announcing Evan's Awesome A/B Tools
1–10 of 26 posts
Re: Announcing Evan's Awesome A/B Tools
#2These look great. For the sake of the permalinks, I'd love to see these hosted on another domain. Maybe a Wizard app domain, to increase brand awareness?
Re: Announcing Evan's Awesome A/B Tools
#3You scooped mine by like 20 minutes, which is weird, so I'll just put this here:
http://www.experimentcalculator.com/
*edit: yours is awesome. Nice work.
Re: Announcing Evan's Awesome A/B Tools
#4Hey Evan,
We met a few times during last years Gig Tank (I am one of the cofounders of http://banyan.co). Awesome to see you killing it. Are you planning on coming back to Chattanooga anytime soon? Would love to grab beers. My email is in my profile, and I would love to reconnect.
Re: Announcing Evan's Awesome A/B Tools
#5You scooped mine by like 20 minutes, which is weird, so I'll just put this here: http://www.experimentcalculator.com/ *edit: yours is awesome. Nice work.
I tried and liked the simplicity of yours! I also learned that as a small ecommerce - 5k/day - you may never have enough visitors to effectively run reliable tests :(
Re: Announcing Evan's Awesome A/B Tools
#6You scooped mine by like 20 minutes, which is weird, so I'll just put this here: http://www.experimentcalculator.com/ *edit: yours is awesome. Nice work.
I tried and liked the simplicity of yours! I also learned that as a small ecommerce - 5k/day - you may never have enough visitors to effectively run reliable tests :(
Depends: if you're small, then you might have low-hanging fruit and bigger conversion increases might be possible. If that happens to be true then measuring things is tractable.
Re: Announcing Evan's Awesome A/B Tools
#7These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :)
As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack:
http://www.thumbtack.com/labs/abba/
It shares the visual component and the linkability, two great features you've nailed. It lacks the live updating and the slider, which is really cool and something I've wanted to add to ABBA for a long time. On the other hand, it supports multiple groups compared against the baseline simultaneously and incorporates a correction for multiple testing into its p-values and confidence intervals, which can be handy. It also uses different mathematics under the hood, but that's not going to be a concern for most users.
Glad to see another step towards a more statistically-aware world!
Re: Announcing Evan's Awesome A/B Tools
#8Evan these are incredible tools, thank you for contributing another brick in the wall for those of us that bleed A/B testing :)
I would love to see Optimizely and VWO embrace similarly non-ambiguous and functional reporting as a default.
EG - just introducing Chi-Squared testing into a discussion with clients or teams that think that they're A/B testing properly by following Optimizely's graphs usually turns the discussion on it's head - "you mean there's a RANGE? well how can we be certain?" etc.
Great work, thank you!
Re: Announcing Evan's Awesome A/B Tools
#9You scooped mine by like 20 minutes, which is weird, so I'll just put this here: http://www.experimentcalculator.com/ *edit: yours is awesome. Nice work.
I tried and liked the simplicity of yours! I also learned that as a small ecommerce - 5k/day - you may never have enough visitors to effectively run reliable tests :(
That's also why experimental psychologists have mixed feelings about running power analyses* to figure out how many participants they'll need in a study to to yield statistically meaningful results--it's almost always a humbling high figure.
Re: Announcing Evan's Awesome A/B Tools
#10These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :) As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack: http://www.thumbtack.com/labs/abba/ It shares the visual component and the linkability, two great…
Small follow-up question: I'm curious how your sample size calculator chooses its value. Traditional power analysis asks for two proportions rather than for a proportion and an effect size. So the sample size for a positive effect differs from that for a negative effect (from the same baseline). I'd imagine your tool would conservatively present the larger of the two. However, it seems to present something near the midpoint of the two. Is that the intention? Or is there some other statistics being used here?
(For example, with an 8% baseline conversion rate, a 1% absolute detectable different, 85% power and 10% significance level, your tool says 10,583 per branch. R's `power.prop.test` gives sample sizes of 11,182 for a positive change (8% vs 9%) and 9,974 for a negative change (8% vs 7%). The exact midpoint is 10,578.)