101–106 of 106 posts
Earlier quoted context omitted.
Yes, and that use case/failure mode should be remembered if the product being pushed overlaps that domain. Is that a large market outside of medical?
A lot of real world cases have imbalanced reward functions. Autonomous vehicles are another example. I don’t doubt regulation systems on spacecraft are similar.
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as you're not using a method designed to be checked at a single point in time. Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects). The type of test conducted in cl…
Earlier quoted context omitted.
Presumably using your method takes longer/requires more samples than a method that only checks once?
I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.
My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better, at best it could be the same.
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a…
Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as you're not using a method designed to be checked at a single point in time. Take a look at clinical trials. Often in clinical trials there are multiple phases, where early stopping is desirable in case the drug has higher-than-expected efficacy (or more-harmful-than-expected side effects). The type of test conducted in cl…
Lotsa things are OK so long as you are doing X and Y etc.
Take a look at a portion clinical trial[1] guidance from FDA. Note specifically the basic Stats guidance:
6.9.1 A description of the statistical methods to
be employed, including timing of any planned
interim analysis(ses).
6.9.2 The number of subjects planned to be
enrolled. In multicenter trials, the numbers of
enrolled subjects projected for each trial site
should be specified. Reason for choice of
sample size, including reflections on (or
calculations of) the power of the trial and
clinical justification.
I fail to see what is so vital about some web site A/B test that one cannot be bothered to think ahead about what defines an observational unit, how many of those one might need to detect an improvement, and wait until after that sample has been attained to test (and, if the web site doesn't get enough visitors to fulfill your sample size requirement for that particular test, that is a different problem entirely).
[1]: https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegula...
Earlier quoted context omitted.
I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.
That's highly counter-intuitive to me. Can you try to motivate why that's the case? My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better , at best it could be the same.
Short answer: in sequential testing you can ask at intermediate stages whether a satisfactory confidence has been reached. If yes you are done and if not you can continue. On average you will hit a 'yes' sooner. For non sequential you cannot do this if you care about correctness (). So the sample size needs to be pessimistic for non-sequential protocols and then you are bound to that commitment.
() If your method ensures correctness even after inspection at intermediate stages then its a sequential method by definition. There is some confusion in literature about Bayesian and sequential. They are orthogonal concepts. Both Bayesian and Frequentist test of hypothesis can be sequential
Earlier quoted context omitted.
That's highly counter-intuitive to me. Can you try to motivate why that's the case? My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better , at best it could be the same.
This is 1940s stuff. Checkout Wald. Short answer: in sequential testing you can ask at intermediate stages whether a satisfactory confidence has been reached. If yes you are done and if not you can continue. On average you will hit a 'yes' sooner. For non sequential you cannot do this if you care about correctness ( ). So the sample size needs to be pessimistic for non-sequential protocols and then you are bound to t…