Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

blog.optimizely.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#41

Post by Silhouette »

gnaritas wrote: post #31
Silhouette wrote: post #28

Earlier quoted context omitted.

> A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result. Well, no, it's a set of numbers with percentage signs after them. Perhaps the documentation for Optimizely specifies how their error margins etc. are derived, but nothing in the linked article does as far as I can see. Without knowing that underlying reasoning, all those pretty graphs and percentages…

What a lazy ignorant statement. If you want to see how they crunch the numbers, go look.

I did. A Google search for

"error bars" site:optimizely.com

turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written.

If you have found other sources where the Optimizely site publicly describes their statistical methodology, please share them. I think several people following this discussion would be interested.

Otherwise, I stand by my earlier comments.

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#42

Post by jules »

meric wrote: post #35
Eliezer wrote: post #34

Dear Optimizely: Your statistics do not tell you that B has a 95% chance of being better than A. Your statistics tell you when the excess of B over A has less than 5% probability, assuming B and A are actually equally effective. A Bayesian would understand this in terms of prior probabilities and likelihood ratios, but to put it into nontechnical terms, suppose that you tried out 15 different alterations and none of…

The p-value represents "Probability this is all luck". So why isn't 1 - p "Probability this isn't all luck"? >> Another extremely important point is that the classical >> statistics you learned to use to decide that something >> was >> decided in advance to do exactly that many trials and >> then stop. I agree. Not doing that I'd say it's "fudging with the numbers". I can't find on their article where they did this t…

The p-value means "chance of this happening, if the two were equally effective". So 1-p means "change of this NOT happening, if the two were equally effective".

Note that in frequentist statistics, the sentence "Probability this is all luck" doesn't even make sense. Either the universe is so that it was luck, or it is not. In frequentist statistics you do not assign probabilities to models of the universe. You only assign probabilities to observations, given a fixed model of the universe.

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#43

Post by gnaritas »

Silhouette wrote: post #41
gnaritas wrote: post #31

Earlier quoted context omitted.

What a lazy ignorant statement. If you want to see how they crunch the numbers, go look.

I did. A Google search for "error bars" site:optimizely.com turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written. If you have found other sources where the Optimizely si…

So ask them; they aren't stupid, they wouldn't be building a business based on A/B testing without using valid methods of testing and displaying results. That you call the results crap because you don't have the perfect details of everything is simply absurd.

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#44

Post by petekoomen »

Eliezer wrote: post #34

Dear Optimizely: Your statistics do not tell you that B has a 95% chance of being better than A. Your statistics tell you when the excess of B over A has less than 5% probability, assuming B and A are actually equally effective. A Bayesian would understand this in terms of prior probabilities and likelihood ratios, but to put it into nontechnical terms, suppose that you tried out 15 different alterations and none of…

[I'm one of the co-founders of Optimizely] Hi Eliezer, thanks very much for your thoughts--these are great. You're absolutely right that classical p-values are not without their shortcomings, which can be significant when misused! A Bayesian approach would address some of these shortcomings, but would also introduce other hurdles, like generalizing the selection of a prior distribution and, more importantly, (IMHO,) distilling likelihood ratios in a way that can be quickly grasped by someone with no formal stats/math background. In simpler terms, classical statistics is not perfect, but in most cases it provides a lot of useful information in an easy-to-grasp way.

That said, we've started looking more seriously at incorporating Bayesian techniques into our interface and would love to get your thoughts on great ways to communicate these concepts to our customers. I've reached out to you directly off-thread and would love chat further!

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#45

Post by Silhouette »

gnaritas wrote: post #43
Silhouette wrote: post #41

Earlier quoted context omitted.

I did. A Google search for "error bars" site:optimizely.com turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written. If you have found other sources where the Optimizely si…

So ask them; they aren't stupid, they wouldn't be building a business based on A/B testing without using valid methods of testing and displaying results. That you call the results crap because you don't have the perfect details of everything is simply absurd.

I didn't call the results "crap". I am simply pointing out that they are meaningless without knowing the methodology behind them. (And we aren't just missing the "perfect details of everything" here. As far as I can see, we have no rigorous details whatsoever.)

I would remind you that you were the person who was attacking another poster's position based on your interpretation of those currently meaningless numbers. It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#46

Post by gnaritas »

Silhouette wrote: post #45
gnaritas wrote: post #43

Earlier quoted context omitted.

So ask them; they aren't stupid, they wouldn't be building a business based on A/B testing without using valid methods of testing and displaying results. That you call the results crap because you don't have the perfect details of everything is simply absurd.

I didn't call the results "crap". I am simply pointing out that they are meaningless without knowing the methodology behind them. (And we aren't just missing the "perfect details of everything" here. As far as I can see, we have no rigorous details whatsoever.) I would remind you that you were the person who was attacking another poster's position based on your interpretation of those currently meaningless numbers. I…

You called them gobbledegook, same difference.

> I am simply pointing out that they are meaningless without knowing the methodology behind them.

Only if you assume incompetence or malice on the part of Optimizely, neither of which you have any valid reason to do. It's perfectly reasonable to assume they aren't stupid and the results are valid.

> It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.

Um, my claim is don't assume they're idiots; that doesn't require me to back anything up.

The poster I replied to wasn't attacking them, he was attacking statistic in general, which is what I was replying to.

Your response was to imply that Optimizely doesn't know what they're doing and therefore their results are invalid until you see how they're crunching the data; that's simply absurd.

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#47

Post by petercooper »

StavrosK wrote: post #39
petercooper wrote: post #26

This is sorta related but since I concluded the test today and this post is here, I thought I'd share. I ran a 5 way split test for 9 days on my newsletter's signup page (JavaScript Weekly). My original page was the 2nd best performing but an identical page just without the subscriber count got a 8% higher conversion (or about 20% more signups in all) with a 90% confidence at the end of testing. The worst performer?…

Were you looking at the statistics all the time to see when they passed 90% confidence? That invalidates them, just so you know. Basically, if you're watching it, it won't science.

I'm not sure how tongue in cheek that was ;-) but 90% confidence was not a goal, at least. I just got bored after a week and wanted to move on. C'est la vie..

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#48

Post by MikeX1555 »

kristofferR wrote: post #9

How does Optimizely compare with Visual Website Optimizer?

I spent the last month using VWO, and despite the website needing a slight Web 2.0 visual update, it has been a good experience.

I'm now getting started with Optimizely and the biggest thing I miss is Video demos. I also liked that VWO had a much smaller JS file with the option to remove Jquery. Fix this Optimizely, I don't need to load Jquery twice!

Re: Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

#49

Post by Pickhardt »

MikeX1555 wrote: post #48
kristofferR wrote: post #9

How does Optimizely compare with Visual Website Optimizer?

I spent the last month using VWO, and despite the website needing a slight Web 2.0 visual update, it has been a good experience. I'm now getting started with Optimizely and the biggest thing I miss is Video demos. I also liked that VWO had a much smaller JS file with the option to remove Jquery. Fix this Optimizely, I don't need to load Jquery twice!

Hey Mike -

Good point about the video demos.

If you have your own version of jQuery running on your site, you can have Optimizely exclude jQuery from the project bundle. Check out this for more info:

http://support.optimizely.com/kb/advanced/does-optimizely-co...

If you have any other questions, you can contact us through support.

- Jeff (with Optimizely)

Post reply on HN