Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Evan's Awesome A/B Tools

evanmiller.org

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Announcing Evan's Awesome A/B Tools

#11

Post by yahelc »

gostevehoward wrote: post #7

These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :) As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack: http://www.thumbtack.com/labs/abba/ It shares the visual component and the linkability, two great…

You'll be interested to hear that the digital analytics team at the Obama campaign sometimes made use of your tool for sharable A/B calculations.

I even built a bookmarklet for quickly grabbing numbers off off of a page (usually for Google Analytics data) and passing them to your calculator. https://github.com/yahelc/ABBA-bookmarklet

Re: Announcing Evan's Awesome A/B Tools

#12

Post by gostevehoward »

yahelc wrote: post #11
gostevehoward wrote: post #7

These are great! Thank you Evan. Your sample size calculator is wonderful and beats the hell out of the 90s tool I've been using which cautions me that my browser must support "JavaScript" to use it :) As an alternative to the Chi-squared calculator, people might want to check out ABBA, a tool I wrote here at Thumbtack: http://www.thumbtack.com/labs/abba/ It shares the visual component and the linkability, two great…

You'll be interested to hear that the digital analytics team at the Obama campaign sometimes made use of your tool for sharable A/B calculations. I even built a bookmarklet for quickly grabbing numbers off off of a page (usually for Google Analytics data) and passing them to your calculator. https://github.com/yahelc/ABBA-bookmarklet

Wow this bookmarklet is awesome! I'm going to start using it. Thanks!

Re: Announcing Evan's Awesome A/B Tools

#14

Post by archildress »

aresant wrote: post #8

Evan these are incredible tools, thank you for contributing another brick in the wall for those of us that bleed A/B testing :) I would love to see Optimizely and VWO embrace similarly non-ambiguous and functional reporting as a default. EG - just introducing Chi-Squared testing into a discussion with clients or teams that think that they're A/B testing properly by following Optimizely's graphs usually turns the disc…

Curious - for someone statistically ignorant like me, can you provide some detail on how we can use Evan's tools for split testing improvements?

I guess the question I'm boiling this down to is... Why are graphs and comparisons of results that Optimizely or VWO produces not good enough?

Re: Announcing Evan's Awesome A/B Tools

#15

Post by timr »

aresant wrote: post #8

Evan these are incredible tools, thank you for contributing another brick in the wall for those of us that bleed A/B testing :) I would love to see Optimizely and VWO embrace similarly non-ambiguous and functional reporting as a default. EG - just introducing Chi-Squared testing into a discussion with clients or teams that think that they're A/B testing properly by following Optimizely's graphs usually turns the disc…

All of the tools used to have this sort of presentation. GWO, in particular, once had a very nice visualization of confidence-range overlaps (the current product blows).

I suspect they removed them for the same reason I tend to avoid the subject when discussing testing with non-technical people: nearly everyone is numerically illiterate, and looking for the "easy" answer. They want a used-car salesman, not a mathematics professor (i.e. "no more of this 'confidence' gobbeldygook -- give me the bottom line").

Sad, but in my experience, nearly universally true.

Re: Announcing Evan's Awesome A/B Tools

#16

Post by WA »

That is really awesome. One suggestion though: Make "relative" the default as well as set "1−β" to 90 or 95%.

I assume that most people have a conversion rate of X, say 30%, and want to increase this by Y, say 20% (30% to 36%). If I consider the type of headline many blog posts and reports have, they are like "How we increased sales, trials, whatever by 50%". That's how they think.

And well, I usually aim for 95% significance.

Re: Announcing Evan's Awesome A/B Tools

#18

Post by aresant »

archildress wrote: post #14
aresant wrote: post #8

Evan these are incredible tools, thank you for contributing another brick in the wall for those of us that bleed A/B testing :) I would love to see Optimizely and VWO embrace similarly non-ambiguous and functional reporting as a default. EG - just introducing Chi-Squared testing into a discussion with clients or teams that think that they're A/B testing properly by following Optimizely's graphs usually turns the disc…

Curious - for someone statistically ignorant like me, can you provide some detail on how we can use Evan's tools for split testing improvements? I guess the question I'm boiling this down to is... Why are graphs and comparisons of results that Optimizely or VWO produces not good enough?

The short answer is that they don't provide much depth.

I've seen Optimizely call something a "winner" with 95% confidence after 48hrs.

The triangular method we use w/the off-the-shelfers is something like:

a) Optimizely base stats

b) Convergence point analysis (useful to correct for day-of-week / unique traffic swings)

c) Chi-Squared Testing which provides a range so that you can actually assess the risk of a high-confidence test. eg look at the example in evan's tool which shows 8.5%-22% and 13.%-28.9%. This means that Sample 1 could be as HIGH as 22.1% conversion and Sample 2 could be as LOW as 13.3%. If this was rated as a high-confidence test that Sample 2 was rated higher than Sample 1 you could be potentially risking a signifigant conversion decrease if you went with Sample 2. EG needs more data and don't just buy into the "this one is better"

Re: Announcing Evan's Awesome A/B Tools

#19

Post by binarysolo »

As a math/stats/data person who doesn't dabble much in web optimization -- can someone explain to me what's awesome about this?

Not to belittle this nice package -- it looks like a basic stats calculator for calculating sample size confidence levels with friendly visualization, and I'm just trying to understand what's being valued on the market/industry right now. Is it because current A/B testing software doesn't provide these basic calculations? Or is it that it's well presented and visualized to a lay crowd?

Re: Announcing Evan's Awesome A/B Tools

#20

Post by bryanh »

Great tools!

My goto reference is still the wonderful btilly presentation about how to A/B test properly, with nice examples and code snippets: http://elem.com/~btilly/effective-ab-testing/

He provided a full on javascript tool that isn't as polished but works great: http://elem.com/~btilly/effective-ab-testing/g-test-calculat...

Post reply on HN