Why Do Models That Predict Failure Fail? [pdf]

Why Do Models That Predict Failure Fail? [pdf]
fdic.gov

A pretty roundabout way of saying that the future was being predicted by extrapolating the past, and that the actual future was unlike the past.

> Interestingly, diversifying the macroeconomic conditions from which the training data are sampled did not generally improve the out-of-time performance of the machine learning methods that we considered.

Mildly interesting indeed, but not quite surprising given that the macroeconomic conditions form a pretty large space. All sets of conditions for which data is available form only a tiny subset of all possible conditions, so increasing the training data from (say) 0.1% to 0.2% of possible inputs is not going to move the needle much.

In this instance in particular (mortgage default prediction), generating better predictions will produce shifts in the very conditions observed. The possibilities for emergent behaviour in a system as complex as the mortgage market are very large.