> A prospective test is blind. You don't know what the data looks like, so you can't cheat.
Suppose I blind myself to the holdout set. What's the difference?
> It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs.
What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not?
Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model?
To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do.
> Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.