Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizely Statistics Engine

optimizely.com

Post reply on HN
Print view View original

21–30 of 57 posts

Re: Optimizely Statistics Engine

#21

Post by dlss »

jmalicki wrote: post #15
mziel wrote: post #12

Earlier quoted context omitted.

Bayesian updating of the posterior? Or if you prefer frequentist algorithms for online learning of classifiers? Not trying to pick a fight, just as a statistitian/ML developer I've seen the same things be reinvented and renamed so many times.

No, if you read their technical paper, it's frequentist sequential testing with false discovery rate control, which is a fairly recent development (I mean, 25 years old is pretty new in statistics). http://pages.optimizely.com/rs/optimizely/images/stats_engin...

I think all OP is trying to point out is that it either agrees with bayesian methods or it's wrong... so at best it's not materially new, and at worst it's using questionable assumptions.

Re: Optimizely Statistics Engine

#22

Post by dargani123 »

darkxanthos wrote: post #10

Every time they say "classic statistics" just insert "what we did before now" and see how frustrated you get with this announcement. The whole point of using them is that people don't need a statistician because the tool should make it easy to run solid tests. That of course hasn't been the case and they're finally admitting it.

Thanks for your comment. This is Darwish, the Product Manager working on Stats Engine. You are correct "classic statistics" is the method we used in the past. It also what is most commonly used in industry (the main reason we started with this method). This was not an easy project for us to take on, but after talking to customers and looking at our historical experiment data, it was clear how important this problem was to solve, and thats why we spent a lot of resources on fixing this. Just for those following along on this comment, its not that "classic statistics" on their own that are incorrect, but rather the misuse of these statistics that can be costly. When used "incorrectly" (not using a sample size calculator, running many goals and variations at a time etc..), you can meaningfully increase your chance of making a bad business decision or commit yourself to unnecessarily long sample sizes. Using statistics correctly is an industry-wide problem that many have tried to solve with education (i.e. give statistics crash courses). We hope that our solution shows how important we think it is that statistics drive day-to-day decisions in organizations and that there are different ways (change the math, not the customer) to get customers to this point. Many companies have data science teams and in-house statisticians that are very aware of these problems, but many don't and thats really where we wanted to help out. You can read more about why we thought this was a serious problem here: http://blog.optimizely.com/2015/01/20/statistics-for-the-int...

Re: Optimizely Statistics Engine

#23

Post by [deleted] »

darkxanthos wrote: post #10

Every time they say "classic statistics" just insert "what we did before now" and see how frustrated you get with this announcement. The whole point of using them is that people don't need a statistician because the tool should make it easy to run solid tests. That of course hasn't been the case and they're finally admitting it.

[deleted]

Re: Optimizely Statistics Engine

#24

Post by blumkvist »

austincav wrote: post #20

I am surprised by all the negative commentary here. On the whole, companies like Optimizely, RJMetrics, Custora, and others are doing more to push statistical analysis to the mass market than anyone else. These tools are not designed for statisticians or ML practitioners so it makes sense they do not put language like Bayesian, etc. front and center. IMO, the more people using data to make decisions, the better.

All optimizely, VWO and other such services provide is WYSIWYG editor and a redirect script. Some pretty (but meaningless) graphs and lots of bullshitting.

More importantly, I'm sure they have people who know that their "A/B tests" most definitely do not work as advertised, so they are misleading their customers on purpose.

Re: Optimizely Statistics Engine

#25

Post by gmu3 »

dlss wrote: post #19
gmu3 wrote: post #18

Earlier quoted context omitted.

I don't think the graph is particularly good either, but I think you're maybe reading it wrong too. The y axis is significance level which makes sense would normally improve as visitors go up. Their line arguably is a moving average regression.

He's saying that as new data arrives, it has to adjust both ways, not only toward the correct answer. (If that wasn't the case, you could just figure out which was the only direction it would move and then stop collecting data. You've already got your answer)

Their graphs don't only adjust up. That one does, but that's because the only really significant downward movement is while their regression is still trailing behind.

Re: Optimizely Statistics Engine

#26

Post by ayy88 »

austincav wrote: post #20

I am surprised by all the negative commentary here. On the whole, companies like Optimizely, RJMetrics, Custora, and others are doing more to push statistical analysis to the mass market than anyone else. These tools are not designed for statisticians or ML practitioners so it makes sense they do not put language like Bayesian, etc. front and center. IMO, the more people using data to make decisions, the better.

It's not that they don't put in 'language like Bayesian', it's a different method. Yes, it is an improvement on the t-test straw-man they mention, but it's less flexible and powerful than Bayesian methods. Once you have a posterior, you can ask different questions that their p-values/confidence intervals don't address. For example, probability of an x% increase in conversion rate, or the risk associated with choosing an alternative. Not too mention multi-armed bandits, which not only are expected to arrive at an answer faster, but also maximize conversions along the way.

Re: Optimizely Statistics Engine

#27

Post by dlss »

austincav wrote: post #20

I am surprised by all the negative commentary here. On the whole, companies like Optimizely, RJMetrics, Custora, and others are doing more to push statistical analysis to the mass market than anyone else. These tools are not designed for statisticians or ML practitioners so it makes sense they do not put language like Bayesian, etc. front and center. IMO, the more people using data to make decisions, the better.

I very much like that people are starting to care about data-driven decisions... However I find it quite aggravating that these tools don't use the best available methods. Optimizely is celebrating that they built a strange, proprietary solution to a very well studied problem.

The situation to me feels a lot like acupuncture, homeopathic medicine, etc. I agree that these doctors and patients have their hearts are in the right place... I just wish they'd channel that energy in a more positive direction. It's frustrating.

Re: Optimizely Statistics Engine

#29

Post by leo_pekelis »

debacle wrote: post #9

This graph is a joke: https://d1qmdf3vop2l07.cloudfront.net/optimizely-marketer-as... And any company trying to sell a statistical tool/package that would actually create a graph like that is selling snake oil. Your model only gets better, digitally, and never sees a regression? And you're using this for web analytics?

Hello, Leo, Optimizely's in-house statistician here. The graph you reference is a schematic to show the differences between Optimizely’s previous statistical platform and Stats Engine. It shows a monotone non-decreasing significance because under our sequential testing framework, the significance value represents the total amount of accumulated evidence against the null hypothesis of no difference between a variation and baseline. This wealth of evidence cannot decrease because you can only get more information about your test as you get more visitors. Of course, it is very possible, as the graph shows, that you will not acquire enough contradictory evidence to reach significance in a reasonable number of visitors. What would have happened if we instead looked continuously at a classical t-test, is the significance would oscillate near the significance threshold. Spurious deviations would cause multiple, contradictory declarations that the test is significant and then not. A savvy A/B tester might wait until the oscillations die down. Sequential testing is a principled, mathematical way to differentiate evidence against the null hypothesis from random oscillations in real time. It should be noted that the chance of a type I error is still controlled at 5%.

You do make a good point that sometimes an A/B test will see regression over time. We have explicitly separated this out because we feel detecting a change in the underlying effect size is different from testing whether the effect is non-zero, and different statistical methods are better suited to one over the other. We have built a policy into our framework that monitors for such temporal effects and signals an A/B test is in a ‘reset’ when we discover them. In our historical database, this happened on about 4% of tests.

I concede all this is a lot to get across in one graph, but we do feel that it is a good representation of how significance behaves under Stats Engine. If you would like to read more about the math behind stats engine, here is a link to a full technical article: http://pages.optimizely.com/rs/optimizely/images/stats_engin...

Re: Optimizely Statistics Engine

#30

Post by aaronjg »

I wrote about the problem with sequential testing in online experiment three years ago on the Custora blog [1]. And Evan Miller wrote about it two years before me on his blog [2]. I'm glad to see Optimizely finally getting on board. Communicating statistical significance to marketers is always challenging, and I'm sure this will lead to better decisions being made.

[1] http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...

[2] http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Post reply on HN