81–90 of 106 posts
Hi, founder of VWO here. We revamped our testing engine to Bayesian in 2015 to prevent the ‘peeking problem’ with frequentists approaches. You can read about our approach https://vwo.com/blog/smartstats-testing-for-truth/
Earlier quoted context omitted.
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…
Hey all, statistician from Optimizely chiming in here. Just wanted to point out that this is exactly the right point. I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple look…
In fact, why use an inferential framework at all (estimating some sort of probability and using it to guide action), rather than directly using a policy learning framework, e.g. modeling this as Q-learning or multi-armed bandit problem?
If at the end of the day you have some objective function (e.g. 'making money'), some known space of actions (e.g. move this widget up the page, change the color, engage with user this way), and a reasonable way to associate those two, then isn't the company literally doing reinforcement learning over time?
It seems one benefit of a reinforcement learning framework is it maintains a set of actions that will still be explored in the future without forcing you to prematurely 'choose' whether A or B is actually better—if A is better in reality, then it will be explored more and more often and B will progressively become downweighted over time.
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as you're not using a method designed to be checked at a single point in time. Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects). The type of test conducted in cl…
Earlier quoted context omitted.
Hey all, statistician from Optimizely chiming in here. Just wanted to point out that this is exactly the right point. I wanted to add one detail--there actually are ways to do early stopping while staying within a frequentist approach. For example, most clinical trials methods are not Bayesian but rather are just fixed-horizon tests that have the allowable amount of Type 1 error "spread out" amongst the multiple look…
What's the tradeoff vs. just taking a direct Bayesian approach? In fact, why use an inferential framework at all (estimating some sort of probability and using it to guide action), rather than directly using a policy learning framework, e.g. modeling this as Q-learning or multi-armed bandit problem? If at the end of the day you have some objective function (e.g. 'making money'), some known space of actions (e.g. move…
Probably because of the maintenance cost of the code that was only explored but never exploited.
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
It’s done in academia too. Along with “remove the outlier data that messes up our p values”
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
Done properly, it might be way more efficient than setting your parameters ahead of time, see [0] ; If I had gotten that response from him, I'd assume that's what he meant.
[0] https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...
Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology
I think that unnecessarily discounts exploratory work. There's nothing wrong with forming a hypothesis after seeing a pattern in data. But remember that it's just a hypothesis -- an unconfirmed guess. After the hypothesis is formulated, then an experiment can be designed to test it, see if its predictions hold up.
Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology
I would say it is. By not being hypothesis driven, you are producing too many degrees of freedom by de facto testing every comparison. This is a huge problem in science too. I have regularly been told to just see what happens and come up with hypotheses after, or others have been unable to say what their hypotheses actually are. Scientists seem less and less likely to be trained in statistics, and in the scientific p…
I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and ru…
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to tre…
If failures essential don't matter (e.g. number of bacteria killed in a petri dish), sure use frequentist p-value.
If failure do matter (e.g. number of patients killed), use Bayesian multi-armed bandit.
Wrote a blog post on it: https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-p...
Earlier quoted context omitted.
I'm not being nitpicky - they are components of a problem of reproducibility, but orthogonal to each other. Bad UI design and a poor backend are both reasons "X website sucks!" but that doesn't mean they're the same. A perfectly designed, un-p-hacked study should still perhaps be held to a stricter p-value criteria than 0.05. And I am correct - because I've done it. Presently working on a paper where, because I prima…
First off, I don’t have any experience with publishing based off the results of simulations. My (short) time in writing papers centered around economics research with observational datasets. I can tell you that given a fixed size dataset, it is not possible to p-hack below a certain threshold in any meaningful way. My colleagues would’ve asked why the gigantic dataset we purchased had 1/4 of its observations thrown o…
This is only true if you haven't collected your own data, and the size of the original sample is known - and that you used all of it. I would suggest that a fixed, known sample size is a relatively rare outcome for many fields.
"Your claim that the threshold and the practice of p-hacking are orthogonal (independent?) is still puzzling to me."
The suggestion is they're unrelated. Changing to say, p = 0.005, will impact studies that aren't p-hacked, and does not p-hacking proof evidence. It potentially makes things more difficult, but not in a predictable and field-agnostic fashion.
"1) Would it take you more time to p-hack a lower threshold, or do all your results yield you a ~0.0000 p-value?"
It might take me more time - but I could also write a script that does the analysis in place and simply stops when I meet a criteria. The question is will it take me meaningfully more time - "run it over the weekend instead of overnight" isn't a meaningful obstacle.
"In simulation based research like yours, it seems to me that even other p-hacking “fixes” like forcing there to be a pre-announcement or sample size, sample structure, etc. wouldn’t address what you say you’re able to do. What can be done to fix it?"
My preference is to move past a reliance on significance testing and report effect sizes and measures of precision at the very least. If one must report a p-value, I'd also require the reporting of the minimum detectable effect size that could be obtained by your sample.
Pre-announcing sample size would...just be a huge pain in the ass, generally.