Optimizely Statistics Engine • Hacker News Board

Re: Optimizely Statistics Engine

#51

Post by leo_pekelis » Tue, Jan 20, 2015, 8:31 PM UTC

calpaterson wrote: post #42

I'm someone that would consider using Optimizely: no formal stats background but understand high school stats, work on web apps professionally and interested in analytics and testing. I've watched the video, read everything on the page and I still don't understand what they're trying to tell me here. Based on my admittedly limited understanding of stats, unless you set the sample size and decide what significance is…

Hey,

If you are interested in learning more about the problems we solved, we lay them out in much more detail in our blog post. It’s a meaty topic and so the post is not short. http://blog.optimizely.com/2015/01/20/statistics-for-the-int...

In a few sentences: in the past if you didn’t use a sample size calculator properly (set a sample size up front and only evaluate test results at that time) and had tests with a lot of goals and variations, you could increase that chance of making an incorrect declaration. With Stats Engine, we allow you to monitor your results real-time and test as many hypothesis as you would like and give an accurate representation of the likelihood your test is actually a winning or losing test. We’re definitely not trying to claim to have invalidated traditional statistics. If you used the proper testing procedure in the past, then Stats Engine will simply give you an easier workflow than before (no need to pick an appropriate sample size, minimum detectable effect, limit the number of hypotheses being tests). Many companies have data-scientists, statisticians, or are otherwise well-informed on the topic, however many are not. Stats Engine allows you to have an accurate Statistical Significance measure without requiring you to set a sample size because it accounts for the errors that are introduced by looking at your results as experiment data comes in.

Re: Optimizely Statistics Engine

#52

Post by leo_pekelis2 » Tue, Jan 20, 2015, 8:48 PM UTC

brownesauce wrote: post #39

Fantastic to see Optimizely changing their stats model. The more education that is done on web experimentation the better as there certainly still is a lot of snake oil being sold out there! Their chosen technique is one way of solving the problem of communicating statistics to non-technical audiences however the interpretation of the results may suffer here. I can imagine that this technique will lead to overestimat…

Glad you like that we’re changing things up!

You do bring up a good point. Even though a sequential test is able to be called much earlier than a fixed horizon test (note this only happens when the effect size is large enough to still guarantee Type I error control), it does not change the fact that estimates of the effect size are more variable when there are fewer visitors. The way we are addressing this is to make confidence intervals more prevalent in our platform. The width of confidence intervals represents our uncertainty in the magnitude of the true effect size with the information currently available. They correctly get more narrow as the experiment goes on as there is increased information from more visitors.

Re: Optimizely Statistics Engine

#53

Post by leo_pekelis2 » Tue, Jan 20, 2015, 8:52 PM UTC

dlss wrote: post #27

austincav wrote: post #20

I am surprised by all the negative commentary here. On the whole, companies like Optimizely, RJMetrics, Custora, and others are doing more to push statistical analysis to the mass market than anyone else. These tools are not designed for statisticians or ML practitioners so it makes sense they do not put language like Bayesian, etc. front and center. IMO, the more people using data to make decisions, the better.

I very much like that people are starting to care about data-driven decisions... However I find it quite aggravating that these tools don't use the best available methods. Optimizely is celebrating that they built a strange, proprietary solution to a very well studied problem. The situation to me feels a lot like acupuncture, homeopathic medicine, etc. I agree that these doctors and patients have their hearts are in…

While our solution is different than the current industry standard in A/B testing platforms, all the techniques we are using have been around in the statistics literature for decades, and are tried and true. The particular sequential test of power one that we use has been around since the 1970s and goes back to the time of Herbert Robbins. And FDR control has been well documented in the past 25 years, most notably by Yoav Benjamini, and Yosef Hochberg. We really are standing on the shoulders of giants.

I think our biggest contribution is presenting a principled, powerful mathematical solution in a way that is accessible to practitioners without a formal statistical background. Even if you do have this knowledge, it’s a chance to use these methods without having to reinvent the wheel every time.

There are various methods which could have been used as solutions, and we looked at many different ones to determine a fit to the user model and experience Optimizely is presenting. We are currently doing an AMA on our community portal and I would be happy to discuss potential solutions or any other comments with you there, https://community.optimizely.com/t5/Product-What-s-New/Ask-m...

Re: Optimizely Statistics Engine

#54

Post by triz » Tue, Jan 20, 2015, 10:42 PM UTC

This is a huge step in the right direction.

Re: Optimizely Statistics Engine

#55

Post by GFK_of_xmaspast » Wed, Jan 21, 2015, 4:18 AM UTC

dlss wrote: post #21

jmalicki wrote: post #15

Earlier quoted context omitted.

No, if you read their technical paper, it's frequentist sequential testing with false discovery rate control, which is a fairly recent development (I mean, 25 years old is pretty new in statistics). http://pages.optimizely.com/rs/optimizely/images/stats_engin...

I think all OP is trying to point out is that it either agrees with bayesian methods or it's wrong ... so at best it's not materially new, and at worst it's using questionable assumptions.

"it either agrees with bayesian methods or it's wrong"

This kind of faith-based statistics is pure ideology.

Re: Optimizely Statistics Engine

#56

Post by yummyfajitas » Wed, Jan 21, 2015, 2:21 PM UTC

ayy88 wrote: post #26

austincav wrote: post #20

I am surprised by all the negative commentary here. On the whole, companies like Optimizely, RJMetrics, Custora, and others are doing more to push statistical analysis to the mass market than anyone else. These tools are not designed for statisticians or ML practitioners so it makes sense they do not put language like Bayesian, etc. front and center. IMO, the more people using data to make decisions, the better.

It's not that they don't put in 'language like Bayesian', it's a different method. Yes, it is an improvement on the t-test straw-man they mention, but it's less flexible and powerful than Bayesian methods. Once you have a posterior, you can ask different questions that their p-values/confidence intervals don't address. For example, probability of an x% increase in conversion rate, or the risk associated with choosing…

You do not want to use a classical bandit for A/B testing. The problem is that most bandit algorithms assume the conversion rate is constant - i.e., saturday and tuesday are the same. If sat and tues have different conversion rates, this will horribly break a bandit.

This is not a theoretical problem. I have a client who wasted months on this.

I know how to fix this (a Bayesian method, BTW), but I haven't published it. As far as I know, there is very little published research into using Bayesian bandits in assorted real world cases like this.

Re: Optimizely Statistics Engine

#57

Post by yummyfajitas » Wed, Jan 21, 2015, 2:23 PM UTC

leo_pekelis wrote: post #49

ayy88 wrote: post #26

Earlier quoted context omitted.

It's not that they don't put in 'language like Bayesian', it's a different method. Yes, it is an improvement on the t-test straw-man they mention, but it's less flexible and powerful than Bayesian methods. Once you have a posterior, you can ask different questions that their p-values/confidence intervals don't address. For example, probability of an x% increase in conversion rate, or the risk associated with choosing…

While I do agree that a sequential hypothesis test like the one we implemented in Stats Engine is different than a completely Bayesian method, I wouldn’t necessarily call it less powerful. In fact, numerous optimality properties exist showing that a properly implemented sequential test minimizes the expected number of visitors needed to correctly reject a null hypothesis of zero difference. I should note that our par…

They make statements on the long-run average lift, which is a quantity that many businesses care about: what will my average lift be if I implement a variation after my A/B test?

Could you state clearly what this guarantee is? Unless I'm making a stupid mistake, such guarantees are impossible even in principle with frequentist statistics.