A cartel of influential datasets are dominating machine learning research
1–10 of 75 posts
Re: A cartel of influential datasets are dominating machine learning research
#2Cartel, really?
Re: A cartel of influential datasets are dominating machine learning research
#3Cartel, really?
Right? I thought so too, but this is the first definition from the American heritage dictionary:
> A combination of independent business organizations formed to regulate production, pricing, and marketing of goods by the members.
And it does seem to apply ¯\_(ツ)_/¯
Re: A cartel of influential datasets are dominating machine learning research
#4Somewhat related to this: What methods do people normally use to measure the quality of a dataset?
For example, if there are 50 datasets of historical weather data how can I determine which one is garbage?
Re: A cartel of influential datasets are dominating machine learning research
#5It takes substantial effort to build a good dataset, proportionally more if it gets bigger, and people like big datasets because you can train more powerful models from them. So I am not surprised that people tend to gravitate towards datasets made by well-funded institutions.
The alternative is either a small dataset that people heavily overfit (eg the MUC6 corpus that was heavily used for coreference at some point where people cared more about getting high numbers than useful results) or things like the Universal Dependencies corpus which are provided by a large consortium of smaller institutions
Re: A cartel of influential datasets are dominating machine learning research
#6Just another example of academic conformity.
Re: A cartel of influential datasets are dominating machine learning research
#7I think they are highly overestimating how much science advanced pattern matching will provide. Certainly no conceptual understanding will ever come from that.
Re: A cartel of influential datasets are dominating machine learning research
#8I think they are highly overestimating how much science advanced pattern matching will provide. Certainly no conceptual understanding will ever come from that.
Advanced pattern matching is called science and has been how humans have made progress by taking data and writing down models for them. Now computers do it. End to end algorithms are 'black box' but machine learning in general is much broader than that and is providing understanding in many fields. Sorry you only see/know the "draw a box around a pedestrian' stuff, but try not to judge entire fields based off of limited exposure.
Re: A cartel of influential datasets are dominating machine learning research
#9I think it's good that well-founded institutions publish well-crafted datasets for everyone else.
It helps small teams develop their model.
The problem here is, though, most publishers don't accept papers if the new proposal isn't backed by benchmarks by the well-known datasets. Even though it can be a competitive approach for a specific field, they reject the paper anyway if it's not performed well on the datasets.
Re: A cartel of influential datasets are dominating machine learning research
#10Too bad they don't cite the paper "The Benchmark Lottery" (M. Dehghani et al, 2021) (https://arxiv.org/abs/2107.07002)
> The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.
Edit: By "they", I was actually referring to the linked article. Strangely, even the paper the article is about does not cite "The Benchmark Lottery" at all.