41–50 of 75 posts
As it turned out, differences between datasets proved significantly larger (by a big margin) than differences between algorithms. And the most popular datasets in fact included biases and eccentricities that were bound to cause such problems.
https://bastibe.github.io/Dissertation-Website/ (figures 12.9-12.11, if you're interested)
Granted, datasets have grown larger and larger over time. This concentration is actually a healthy sign of a community maturing and converging towards common benchmarks.
These datasets are open to all for research and have fueled considerable progress both in academia and industry.
The authors would be well-advised to look at what is happening in the community. Excerpts from the program of NeurIPS happening literally this week:
Panels: - The Consequences of Massive Scaling in Machine Learning - The Role of Benchmarks in the Scientific Progress of Machine Learning - How Copyright Shapes Your Datasets and What To Do About It - How Should a Machine Learning Researcher Think About AI Ethics?
All run by top-notch people coming from both academia and industry, and from a variety of places in the world.
I am not saying that everything is perfect, but this article paints a much darker picture than needed.
Oh boy. PWC is not even close to a representative sample of what datasets are being used in papers. It's also often out of date.
This is what my dissertation was about, in a different field: Everyone using the same datasets, even though they are severely flawed. As it turned out, differences between datasets proved significantly larger (by a big margin) than differences between algorithms. And the most popular datasets in fact included biases and eccentricities that were bound to cause such problems. https://bastibe.github.io/Dissertation-Webs…
The pay2publish-paper was sent back by 2 journals with reviews indicating exactly "our method is better, just use the right (our) thing" (which I _all_ refuted for the professor by doing it, but the editor wouldn't hear anymore...). And then there's papers predicting features through a complex preprocessing pipeline in these journals. Academia and big companies are just idiotic.
Earlier quoted context omitted.
Is the term "browbeat" fair here? I don't think they're making calls and saying "Oh, nice paper, but I notice you used this other dataset..." No, they're putting out good quality datasets that people want to use. If that earns them a citation, good for them. Its the least I can do for helping me test my algo.
It depends, feedback is rarely as polite as "I noticed you used this other dataset". The feedback would probably look like. - "Nice paper, however the results are not relevant to current research due to the use of X dataset rather than Y or Z datasets score 2/5 do not accept." - "Nice paper, however the results are of unknown quality due to the use of X dataset 3/5 recommend poster track". In fact I'd generally say t…
Earlier quoted context omitted.
That's true of all organizations that release data. They're regulating their own data. They're not regulating the use of that data
Since we're talking about this in a definitional context you can just as easily argue that a drug cartel doesn't regulate the use of their drugs..
A little confused: Are these big "dominant" institutions charging for this data? No, the spend a lot of resources putting them together and give them away free. Are they preventing others from giving away data? No, but it costs a lot and they bear that cost. Are they forcing smaller institutions to use their data? No. Its just a free resource they offer. Do they get the grants themselves because they have some kind o…
It's well known that neural networks can easily inherit biases from their training data. It's also well known that datasets generated by western universities are widely used in training and evaluating neural networks.
If my training set is full of pictures of Stanford CS undergraduates, I could end up with a computational photography system that makes everyone look like Stanford CS undergraduates, or a historical photo colourisation system that makes everyone look like Stanford CS undergraduates, or a self driving car pedestrian tracking system that expects 90% of pedestrians to look like Stanford CS undergraduates.
And if the people who make the model say "Hey, our model's biases aren't our responsibility, we're just representing the training data as best we can" and the people who make the training data say "Hey, we never claimed it was perfect, you can take it or leave it" these problems might fall through the cracks.
Earlier quoted context omitted.
Seems like a distinct issue to me, though there are obvious parellels and similar outcomes.
Because AIs are relatively narrowly focused, they suffer greatly from limited data sets. More often than not, an AI will simply memorize test data rather than "learning". This impacts the benchmarking process, so I think it's relevant.
When someone used the word “overfitting” I usually take that to mean that a model has begun to enter the phase where further improvement is leading to lower generalization. In fact, as far as I can tell, we are not overfitting in this sense. When I have seen papers examine whether progress on, let’s say, imagenet, actually generalizes to other categorization datasets the answer is yes. What we have been seeing is tha…
There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse.
https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811
There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses has only 10,000 test examples. Performance on these held-out examples is, unsurprisingly, a bit worse:
When someone used the word “overfitting” I usually take that to mean that a model has begun to enter the phase where further improvement is leading to lower generalization. In fact, as far as I can tell, we are not overfitting in this sense. When I have seen papers examine whether progress on, let’s say, imagenet, actually generalizes to other categorization datasets the answer is yes. What we have been seeing is tha…