Calculating the sample size required for developing a clinical prediction model
1–10 of 22 posts
Re: Calculating the sample size required for developing a clinical prediction model
#2This is part of the reasons why A/B testing might fail in reality, it is not just about collecting random number of responses about A and B.
When doing a test you should plan it in advance based on other data, you should execute the test as planned and analyze the results accordingly failing any of the tests will results in wrong to totally wrong conclusions.
Re: Calculating the sample size required for developing a clinical prediction model
#3This is part of the reasons why A/B testing might fail in reality, it is not just about collecting random number of responses about A and B. When doing a test you should plan it in advance based on other data, you should execute the test as planned and analyze the results accordingly failing any of the tests will results in wrong to totally wrong conclusions.
A concrete example:
For starters, consider this xkcd comic: https://xkcd.com/882/
If you're doing A/B testing where you continuously watch the results, and conclude your test as soon as you see the numbers significantly deviate, what you're doing is more-or-less equivalent to what's happening in the comic, only it's the number of jelly beans that you vary instead of their color. Not exactly the same, since in the numbers case your draws from the random variable aren't independent, but still.
Re: Calculating the sample size required for developing a clinical prediction model
#4The link is not to the paper itself but to a response to the paper, this might be (is probably?) intentional but the title is a bit confusing.
Essentially the response, submitted by several ML / CS / math researchers, addresses a note in the paper which recommends against train/test split in model training, calling it "inefficient". The response is dedicated to explaining what generalization error is and why estimating it is important, and how that's basically impossible without any sort of train/test split or cross-validation.
Re: Calculating the sample size required for developing a clinical prediction model
#5The link is not to the paper itself but to a response to the paper, this might be (is probably?) intentional but the title is a bit confusing. Essentially the response, submitted by several ML / CS / math researchers, addresses a note in the paper which recommends against train/test split in model training, calling it "inefficient". The response is dedicated to explaining what generalization error is and why estimati…
The article is there, if you click on the "Article" tab.
Perhaps if any staff are paying attention on the holiday, we could get the "/rr" chopped off the end of the link, so that we get to the main article instead?
Re: Calculating the sample size required for developing a clinical prediction model
#6This is part of the reasons why A/B testing might fail in reality, it is not just about collecting random number of responses about A and B. When doing a test you should plan it in advance based on other data, you should execute the test as planned and analyze the results accordingly failing any of the tests will results in wrong to totally wrong conclusions.
> When doing a test you should plan it in advance based on other data
I think this is not entirely true. In areas where experimentation feedback is fast (and experiments are probably cheaper to run), the problem is much more accurately and sample efficiently solved by Thompson Sampling [1,2] which in fact dictates you to have a broad enough prior over the solution space and then let the posterior dictate your conclusions.
[1]: https://www.microsoft.com/en-us/research/wp-content/uploads/...
Re: Calculating the sample size required for developing a clinical prediction model
#7The link is not to the paper itself but to a response to the paper, this might be (is probably?) intentional but the title is a bit confusing. Essentially the response, submitted by several ML / CS / math researchers, addresses a note in the paper which recommends against train/test split in model training, calling it "inefficient". The response is dedicated to explaining what generalization error is and why estimati…
The article is there, if you click on the "Article" tab. Perhaps if any staff are paying attention on the holiday, we could get the "/rr" chopped off the end of the link, so that we get to the main article instead?
My guess is that OP intended to submit the response, I was really surprised to see the note in the article and found the response interesting (even the fact of its existence). But that's just my take :)
Re: Calculating the sample size required for developing a clinical prediction model
#8And if you’re a Bayesian, you can choose a stopping criterion based on your desired degree of certainty. This can have practical benefits of not having to run an experiment to its end.
A family member was part of the control group for a cancer treatment study a few years back. The study chose a stopping criterion based on Bayesian methods. Relatively early into the study they were able to determine that it made sense to move people from the control group to the treatment group.
Re: Calculating the sample size required for developing a clinical prediction model
#9And if you’re a Bayesian, you can choose a stopping criterion based on your desired degree of certainty. This can have practical benefits of not having to run an experiment to its end. A family member was part of the control group for a cancer treatment study a few years back. The study chose a stopping criterion based on Bayesian methods. Relatively early into the study they were able to determine that it made sense…
You don't have to be a bayesian to do that.
Once you're dealing with real people, early termination is a huge deal, and it quickly goes beyond just the interpretation of statistics. You almost always want to have 3rd party referees involved anyways.
Re: Calculating the sample size required for developing a clinical prediction model
#10Estimating a minimum required sample size is one of the most common questions asked by clinical or biomedical collaborators before embarking on a research project. This is especially true when ML is an option. This paper provides rules of thumb and a digestible amount of theory that could inform such conversations, and will surely become a popular reference.
Note intuition from traditional statistics does not universally apply to deep learning and/or extremely high-dimensional data. For example, deep neural networks with 1-4 orders of magnitude more parameters than training examples can still generalize well to unseen data.