31–40 of 57 posts
Earlier quoted context omitted.
I believe they're referring to the field of statistics as a unit instead of a collection of multiple statistics.
Yes, but it reads badly.
Every time they say "classic statistics" just insert "what we did before now" and see how frustrated you get with this announcement. The whole point of using them is that people don't need a statistician because the tool should make it easy to run solid tests. That of course hasn't been the case and they're finally admitting it.
Thanks for your comment. This is Darwish, the Product Manager working on Stats Engine. You are correct "classic statistics" is the method we used in the past. It also what is most commonly used in industry (the main reason we started with this method). This was not an easy project for us to take on, but after talking to customers and looking at our historical experiment data, it was clear how important this problem w…
Earlier quoted context omitted.
I don't think the graph is particularly good either, but I think you're maybe reading it wrong too. The y axis is significance level which makes sense would normally improve as visitors go up. Their line arguably is a moving average regression.
He's saying that as new data arrives, it has to adjust both ways, not only toward the correct answer. (If that wasn't the case, you could just figure out which was the only direction it would move and then stop collecting data. You've already got your answer)
This graph is a joke: https://d1qmdf3vop2l07.cloudfront.net/optimizely-marketer-as... And any company trying to sell a statistical tool/package that would actually create a graph like that is selling snake oil. Your model only gets better, digitally, and never sees a regression? And you're using this for web analytics?
Hello, Leo, Optimizely's in-house statistician here. The graph you reference is a schematic to show the differences between Optimizely’s previous statistical platform and Stats Engine. It shows a monotone non-decreasing significance because under our sequential testing framework, the significance value represents the total amount of accumulated evidence against the null hypothesis of no difference between a variation…
Interesting that Optimizely is positioning themselves as the over arching discipline as "Statistics reinvented for the internet age". My guess is to parry against the onslaught of A/B testing and optimization platforms for web and mobile from all directions. Of course with their stable of PHD statisticians and data scientists, Optimizely is the answer.
Earlier quoted context omitted.
Thanks for your comment. This is Darwish, the Product Manager working on Stats Engine. You are correct "classic statistics" is the method we used in the past. It also what is most commonly used in industry (the main reason we started with this method). This was not an easy project for us to take on, but after talking to customers and looking at our historical experiment data, it was clear how important this problem w…
What's so particularly embarrassing is that you clearly did not have any competent statisticians on board until now. This was not some big surprise that needed "a lot of resources to fix." This is something that should be obvious to anyone who understands hypothesis testing, and is something that statisticians have been describing how to do correctly for over 50 years: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1551…
Earlier quoted context omitted.
What's so particularly embarrassing is that you clearly did not have any competent statisticians on board until now. This was not some big surprise that needed "a lot of resources to fix." This is something that should be obvious to anyone who understands hypothesis testing, and is something that statisticians have been describing how to do correctly for over 50 years: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1551…
By that logic, no one else in the industry (Adobe, Mixpanel, VWO) has competent statisticians. That's silly.
This graph is a joke: https://d1qmdf3vop2l07.cloudfront.net/optimizely-marketer-as... And any company trying to sell a statistical tool/package that would actually create a graph like that is selling snake oil. Your model only gets better, digitally, and never sees a regression? And you're using this for web analytics?
Hello, Leo, Optimizely's in-house statistician here. The graph you reference is a schematic to show the differences between Optimizely’s previous statistical platform and Stats Engine. It shows a monotone non-decreasing significance because under our sequential testing framework, the significance value represents the total amount of accumulated evidence against the null hypothesis of no difference between a variation…
> if we instead looked continuously at a classical t-test, is the significance would oscillate near the significance threshold
So there's your answer: the y-axis on the chart has an unlabeled different meaning for the blue line.
While I have you here Leo, can you explain why you would want to chart only the accumulated evidence for X? It's meaningless without knowing how much evidence has been accumulated for not X.
Their chosen technique is one way of solving the problem of communicating statistics to non-technical audiences however the interpretation of the results may suffer here. I can imagine that this technique will lead to overestimations of the effect size in situations where the threshold is reached early in an experiment as it will reward extreme values observed when the experiment is under powered.
"95% chance you'll make the right decision" vs "30% chance you'll make the wrong decision", emphasis mine.
