11–20 of 22 posts
And if you’re a Bayesian, you can choose a stopping criterion based on your desired degree of certainty. This can have practical benefits of not having to run an experiment to its end. A family member was part of the control group for a cancer treatment study a few years back. The study chose a stopping criterion based on Bayesian methods. Relatively early into the study they were able to determine that it made sense…
And if you’re a Bayesian, you can choose a stopping criterion based on your desired degree of certainty. This can have practical benefits of not having to run an experiment to its end. A family member was part of the control group for a cancer treatment study a few years back. The study chose a stopping criterion based on Bayesian methods. Relatively early into the study they were able to determine that it made sense…
You don't have to be a bayesian to do that. Once you're dealing with real people, early termination is a huge deal, and it quickly goes beyond just the interpretation of statistics. You almost always want to have 3rd party referees involved anyways.
The reasoning for doing so is much more coherent (imo) in a Bayesian framework where you don't have to say that you are "going beyond" statistics.
Earlier quoted context omitted.
You don't have to be a bayesian to do that. Once you're dealing with real people, early termination is a huge deal, and it quickly goes beyond just the interpretation of statistics. You almost always want to have 3rd party referees involved anyways.
> You don't have to be a bayesian to do that. The reasoning for doing so is much more coherent (imo) in a Bayesian framework where you don't have to say that you are "going beyond" statistics.
Some of the frequentist approaches to early-stopping seem pretty coherent to me. Curtailiment, for example, just stops collecting data once it will no longer change the outcome of a test. The frequentist emphasis on error control for the procedure also seems like a reasonable fit for a regulatory regime where you're actually making decisions (no argument that it is...weirder for basic research)
At any rate, some approaches for early-stopping have sensible frequentist and Bayesian properties, which is nice.
It's related to Goodheart's law: "When a measure becomes a target, it ceases to be a good measure".
It also works in one's personal life. If you find that you made a mistake, don't just fix that particular thing, go back and see what other similar things may need fixing. It connects to the idea of fixing the deeper reason than fixing the symptoms. "Deeper reason" just means, something that generalizes.
By the way, you need to be careful about your data split. This always depends on how you intend to use the model. If you intend to use it on new, unseen patients, then the train and test data cannot overlap with respect to people. Another obvious case is videos: You can't just take every even frame as a training sample and every odd frame as test. Even though technically you would be testing on non-overlapping data, the performance measure would be biased compared to real world performance. Or if you want to classify burglars from security camera footage, you may need to test it on new camera setups from different houses if you intend to deploy it to new houses. If your scenario is such that you'd perform training on each new site and run a location specific model, you can test on images from the same site.
You always have to use your brain to decide what to do.
Earlier quoted context omitted.
> You don't have to be a bayesian to do that. The reasoning for doing so is much more coherent (imo) in a Bayesian framework where you don't have to say that you are "going beyond" statistics.
By "beyond statistics", I think they meant that trials need on-going monitoring for things beyond the outcome variable itself: safety, data quality, and even feasibility of completing the trial itself. Once you've got that infrastructure in place, interim monitoring of the outcome isn't much extra work. Some of the frequentist approaches to early-stopping seem pretty coherent to me. Curtailiment, for example, just st…
Agreed that things are weirder for basic research.
Earlier quoted context omitted.
The article is there, if you click on the "Article" tab. Perhaps if any staff are paying attention on the holiday, we could get the "/rr" chopped off the end of the link, so that we get to the main article instead?
My guess is that OP intended to submit the response, I was really surprised to see the note in the article and found the response interesting (even the fact of its existence). But that's just my take :)
Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test exam…
This is a great point, and it articulates some challenges that I’ve seen with test-driven development.
I would say that if you can write tests that fully specify the desired functionality, rather than merely check a few possible inputs, it’s less of an issue. This is a reason to try to build things with less knobs to twiddle, so the space of inputs has lower dimensionality.
Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test exam…
Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test exam…
> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test ex…
Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test exam…