Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A cartel of influential datasets are dominating machine learning research

unite.ai

Post reply on HN
Print view View original

71–75 of 75 posts

Re: A cartel of influential datasets are dominating machine learning research

#71

Post by throwawaygh »

cs702 wrote: post #69

The title is linkbait : There is no cartel. The word cartel implies collusion. There is no collusion. Using the word "cartel" in this case is not only wrong, but also... insulting to all the hardworking individuals who have toiled away in obscurity to compile, clean-up, label, package, and publish influential datasets. Despite the terrible title, the OP is worth a read. It summarizes a recent research paper, also wor…

> Surely we can find ways to address the winner-take-all dynamics of benchmark popularity

Reviewer 2 is probably the only/best way to address this. More-so if they're a grant reviewer, of course, but paper reviewers will do ok as well.

Re: A cartel of influential datasets are dominating machine learning research

#72

Post by whimsicalism »

ansgri wrote: post #67
whimsicalism wrote: post #43

> Among their findings – based on core data from the Facebook-led community project Papers With Code (PWC) Oh boy. PWC is not even close to a representative sample of what datasets are being used in papers. It's also often out of date.

When you cannot analyze every paper ever published you need to have some relevant criterion for inclusion -- and PWC is somewhat of a GitHub of AI science (at least in early days of it) -- there may be multiple collections, but PWC is by far the most accessible and thus reasonably captures current and emerging trends some dataset-heavy research fields.

But because the effort of adding a new dataset is more than a new paper, this dataset will systemically overestimate the number of papers that include the most popular datasets.

Re: A cartel of influential datasets are dominating machine learning research

#73

Post by sdenton4 »

dontreact wrote: post #68
sdenton4 wrote: post #57

Earlier quoted context omitted.

>> "This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down." This is a problem that is more common for classification problems, I think. Generative and self-supervised models (trained with augmentation) tend to just get better forever (with some asymptote) because memorization isn't a viable strategy. I personally think image classification…

Nearly all useful machine learning is supervised, still. And if you are using a neural network, it will eventually memorize. This is fine though, we have early stopping :)

Eh, I work in audio ml, where most of the interesting and useful work is in conditioned generative models. Compression, TTS, source separation. The bias towards classification is a side effect of people starting at imagenet too long; I really think it's holding the field back.

Re: A cartel of influential datasets are dominating machine learning research

#74

Post by zibzab »

dunefox wrote: post #24
zibzab wrote: post #4

Somewhat related to this: What methods do people normally use to measure the quality of a dataset? For example, if there are 50 datasets of historical weather data how can I determine which one is garbage?

I would say it's garbage if there's a paper like this: https://arxiv.org/abs/1902.01007

That looks interesting.

Any more papers on the subject that you can recommend?

Re: A cartel of influential datasets are dominating machine learning research

#75

Post by dunefox »

zibzab wrote: post #74
dunefox wrote: post #24

Earlier quoted context omitted.

I would say it's garbage if there's a paper like this: https://arxiv.org/abs/1902.01007

That looks interesting. Any more papers on the subject that you can recommend?

At the moment, no, but this could be helpful: https://www.connectedpapers.com/main/42ed4a9994e6121a9f325f5...
Post reply on HN