When someone used the word “overfitting” I usually take that to mean that a model has begun to enter the phase where further improvement is leading to lower generalization.
In fact, as far as I can tell, we are not overfitting in this sense. When I have seen papers examine whether progress on, let’s say, imagenet, actually generalizes to other categorization datasets the answer is yes.
What we have been seeing is that the slope of this graph is flattening out a bit. Whereas in the past a 1% improvement on imagenet would have meant a 1% improvement on a similarly collected dataset, nowadays it will be more like .5% (not exact numbers just using numbers to illustrate what I mean by diminishing returns.)
If an institution or a lab can show that progress on their dataset -better- predicts the progress on a bunch of other closely related tasks, then as researchers become convinced of that, they will switch over. Right now there isn’t a great alternative because it’s not easy to create such a dataset. Scale is critical.
Imagenet really was on the right track as far as collecting images of nearly every semantic concept in the English language. So whatever replaces it will have to be similarly thorough and broad.
In my opinion the biggest weakness of currently existing datasets is that they are typically labeled once per image with no review step. So I think the answer here isn’t
“Let’s get researchers to use smaller datasets from smaller institutions”
It would be more like
“We have to figure out a way to get a bigger, cleaner version of existing datasets and then prove that progress on those datasets is more meaningful”
The realistic way this plays out is that some institution in the “cartel” releases a better dataset and then lots of small labs try it out and show that progress on that dataset better predicts progress in general.