Earlier quoted context omitted.
Cross validation doesn't solve that problem. As the Wikipedia article says: "The variance of F* can be large.[26][27] For this reason, if two statistical procedures are compared based on the results of cross-validation, the procedure with the better estimated performance may not actually be the better of the two procedures (i.e. it may not have the better value of EF). Some progress has been made on constructing conf…
Well the historical data {(x, y)...} is assumed to be distributed according to the true distribution, such that y = t(x) where t is the true function which maps x to y. Of course, in many situations, no such function exists (ie., there are genuinely ambigious xs, such that t(x) cannot produce a single y -- consider an ambigous cat/dog picture). If we sweep models f1,...fn across the validation set ... and choose max(…
Re: ‘Simple’ AI can anticipate bank managers’ loan decisions to over 95% accuracy
#91Thank you for the detailed response. I think this is exactly what I worry about. If you were cynical enough and had enough datasets x1...xn and models f1...fn you would eventually a dataset on which a simple model performs well and be able to publish a paper like this. Even if the author here wasn't cynical enough to do that on purpose, many individually well meaning researchers looking for a nice result to publish effectively do the same thing!