71–80 of 106 posts
Earlier quoted context omitted.
Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
Earlier quoted context omitted.
If you’ve taken some statistics or econometrics, you’ve probably heard of “significance levels” and “p-values”. For some reason, academia choose 0.05 as a threshold for “meaningful” or “significant” results. Generally, a 0.05 p-value means that you would observe your result in 5% of experiments due to random sampling error. I.e. if I tested “is X correlated with cancer”, and my null hypothesis is “X isn’t correlated…
The problem with 0.05 isn't how lenient it is, but rather the fact that a default exists at all. A p-value should be chosen (before running the experiment!) based on how confident the researcher wants to be in their result. For my high school statistics final project, I did an experiment to test whether a stupid prank/joke was funny. Had a pretty terrible experimental design (tons of bias) and tiny sample size ( That…
I agree that over reliance on a single metric, like a p-value, gets us Goodhart’s Law type problems.
In econometrics, and really any other statistics adjacent field, if you’ve correctly estimated your standard errors, and are using something like https://en.m.wikipedia.org/wiki/Newey%E2%80%93West_estimator where appropriate, there is nothing wrong with using a p-value as a general approximation of significance.
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
It's perfectly fine to run an experiment until you get a significant result. You just have to do the maths differently - that's what most people don't know.
That is easier than explaining Bayesian methods to people who cannot handle classical Stats.
Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?
We set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers i…
Beysian results aren't p-values, a 60-70% probably that treatment beats control is just that, not a pvalue of .4 or .3 (which would say nothing).
Earlier quoted context omitted.
>With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences. Given the real-time nature of web experiments, Bayesian design seems like the way forward. Completely agree with Evan Miller here, thank you for sharing the link.
Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
Suppose you're Facebook and you decide to test a new landing page on 1 million users. You roll out the test and notice that after 10,000 users, the new page is killing engagement. Whoops, turns out it had a bug and isn't even loading. Even though no medical patients are dying, this is still a very negative outcome for Facebook. Obviously they shouldn't test on 990,000 more users before fixing the bug, but that's what slavish adherence to pre-registered trial lengths would tell you to do, because it's 'cheating' to notice that there's a problem after the first 10,000 tests.
Some of you might say, sure, for extenuating circumstances like a bug you can break procedure. But in that case I think the logic slides down a slippery slope. What if in the example above instead of a bug we just had a feature performing terribly? In either case, the right move is to end the test early, since you don't need all the data points to measure a strong signal.
We were excited to collaborate with the authors on this study. Keep in mind the data used in this analysis is from 2014, before we introduced sequential testing and FDR correction as to specifically address this p-hacking issue. I expect these results are in line with any platform using fixed-horizon frequentist methods.
Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects).
The type of test conducted in clinical trials explicitly allow for multiple looks while maintaining correct control of the Type 1 error rate. At Optimizely we essentially have a version of this where the monitoring can be conducted contiuously with rigorous control of Type 1 error.
Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…
I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple looks that are planned in advance.
At Optimizely we essentially have a continuous version of this that does in fact allow for multiple looks with rigorous control of Type 1 error. As tedsanders mentions, the key upside is that if you start an experiment with a larger-than-expected lift, you can terminate it early. Then over many repeated experiments, you gain a lot in terms of average time to significance.
The dissonance in this discussion mostly stems from the fact that this paper (which we actually collaborated on!) uses data from 2014, before we rolled out this new Stats Engine.
For more, I would encourage a look at our paper: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
Earlier quoted context omitted.
Except for your example of patients being catastrophically harmed. In that case, false negatives and false positives are not equally undesirable.
Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
https://www.nytimes.com/2014/06/30/technology/facebook-tinke...
Earlier quoted context omitted.
I'm curious about the wording "effects are truly null". I was always under the impression that you can never really "accept the null", but rather "fail to reject the alternative".
In your standard NHST test, sure. But you can do different models. If I'm reading OP right, what they do is a mixture model, in which effects are assumed to come either from a zero-mean distribution or a positive-mean distribution with unknown probability _P_/1-_P_ and then you fit the collection of >2k effect sizes to find out what value of _P_ best fits the dataset as a whole. Apparently the best fit assumes that ~…