Earlier quoted context omitted.
The simple idea is to do stuff like train your model on randomized subsets of your data and then compare its performance to using all the data you have. This doesn't overcome a true "black swan", but that's not what NN are meant to be doing anyhow.
"The simple idea is to do stuff like train your model on randomized subsets of your data and then compare its performance to using all the data you have." How do you do this when you cannot verify that your data, in subset or in whole, is accurate? And furthermore you don't know how inaccurate it is?
You train on a subset of the initial data. Even if the data has a certain number of incorrect frames, it should still do a decent job getting a lot of things right.
Then you manually loop through all the images of the data set for which the network has detected something that isn't present in the annotations (and vice versa). If the network correctly identified a missing item that wasn't in the original set, all you need to do is press "correct" (and, again, vice versa). You now have an improved data set.
Retrain, rinse, repeat.
Eventually, you'll converge to a case where you have consistency between training and annotations. And then, you manually go through all images again to weed out the final mistakes.
The benefit of this method is that it's much faster to click "correct" that it is to draw rectangles on the screen to label something.