Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Concept of a marketplace for machine learning datasets

datapie.in

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Show HN: Concept of a marketplace for machine learning datasets

#21

Post by minimaxir »

jamesblonde wrote: post #17
minimaxir wrote: post #7

There are a few services popping up with aim to provide data repositories for analysis/ML (Kaggle, data.world, /r/datasets) As someone who likes making analyses from random datasets, I have a few issues with these types of services: 1) There is often no indication of the distribution rights of the data, or whether the data was obtained ethically from the source (i.e. following the ToS). I made this mistake when I use…

Good points. We are soon going to release a p2p system for sharing datasets, backed by Hadoop clusters. You install the Hadoop stack (localhost or distributed), then you can free-text search for datasets that have been made 'public' on any hadoop cluster that participates in the 'ecosystem'. We expect it to be self-policing, but there will be a way to report illegal distribution of datasets. The solution is based on…

That only addresses issue #3, which is much less important than issues #1 and #2. (and arguably worse: decentralization makes proper sourcing harder)

Re: Show HN: Concept of a marketplace for machine learning datasets

#23

Post by akshaynathr »

huac wrote: post #20
akshaynathr wrote: post #11

Earlier quoted context omitted.

Yeah. I was thinking of a crowdsourced marketplace platform which follows a particular standard so all dataset supports major ML platforms, API support for programming languages for preprocessing huge datasets etc.People can prepare and sell /buy datasets. This can bring people from various streams together to solve really major issues using the data.

what is a 'particular standard for data'

Many datasets available online does not clearly tell the quality of data. So I think, having a particular standard of quality can really help.

Re: Show HN: Concept of a marketplace for machine learning datasets

#24

Post by bussiere »

jamesblonde wrote: post #17
minimaxir wrote: post #7

There are a few services popping up with aim to provide data repositories for analysis/ML (Kaggle, data.world, /r/datasets) As someone who likes making analyses from random datasets, I have a few issues with these types of services: 1) There is often no indication of the distribution rights of the data, or whether the data was obtained ethically from the source (i.e. following the ToS). I made this mistake when I use…

Good points. We are soon going to release a p2p system for sharing datasets, backed by Hadoop clusters. You install the Hadoop stack (localhost or distributed), then you can free-text search for datasets that have been made 'public' on any hadoop cluster that participates in the 'ecosystem'. We expect it to be self-policing, but there will be a way to report illegal distribution of datasets. The solution is based on…

Will you make a ranking system for noting user and dataset ?

I recommand using true skill for noting user and dataset.

Keep up the good job.

Re: Show HN: Concept of a marketplace for machine learning datasets

#25

Post by whyileft »

minimaxir wrote: post #7

There are a few services popping up with aim to provide data repositories for analysis/ML (Kaggle, data.world, /r/datasets) As someone who likes making analyses from random datasets, I have a few issues with these types of services: 1) There is often no indication of the distribution rights of the data, or whether the data was obtained ethically from the source (i.e. following the ToS). I made this mistake when I use…

Is there actual case law that prohibits the use of copyrighted material for corpora and other training data?

Sure distribution can have issues, but do you have any references for simple possession as training and test data?

Re: Show HN: Concept of a marketplace for machine learning datasets

#26

Post by EternalData »

There's probably some utility to it: a lot of problems involve hacking together datasets, sometimes in dubious ways. There's also value, especially for startups that are looking to build simple neural net applications (ex: identifying plates of food from different restaurants) which are very data-dependent. Researchers may also want to reflect the cost assembling datasets (ex: MTurk, processing power) and open up datasets that may never have been open before.

My general sense on this though is that I'd like there to be more of an incentive for people to open up their datasets to the larger public. Maybe I'm being idealistic but a crowdsourcing type function where you pay for X dataset together with other users and then it's released under MIT, forever free etc.

As others have mentioned that'll probably bump against usage rights issues, a larger problem you'll have to deal with independent of your need to sell or distribute the datasets in question.

Re: Show HN: Concept of a marketplace for machine learning datasets

#27

Post by Cacti »

jamesblonde wrote: post #17
minimaxir wrote: post #7

There are a few services popping up with aim to provide data repositories for analysis/ML (Kaggle, data.world, /r/datasets) As someone who likes making analyses from random datasets, I have a few issues with these types of services: 1) There is often no indication of the distribution rights of the data, or whether the data was obtained ethically from the source (i.e. following the ToS). I made this mistake when I use…

Good points. We are soon going to release a p2p system for sharing datasets, backed by Hadoop clusters. You install the Hadoop stack (localhost or distributed), then you can free-text search for datasets that have been made 'public' on any hadoop cluster that participates in the 'ecosystem'. We expect it to be self-policing, but there will be a way to report illegal distribution of datasets. The solution is based on…

Could you explain what problem you're trying to solve here? Are there really that many researchers who have access to modern (and expensive) GPU hardware that don't have bandwidth or disk space available? Or are there many researchers who are putting in lots of time assembling a dataset but don't have the bandwidth to distribute it?

Re: Show HN: Concept of a marketplace for machine learning datasets

#28

Post by jamesblonde »

minimaxir wrote: post #21
jamesblonde wrote: post #17

Earlier quoted context omitted.

Good points. We are soon going to release a p2p system for sharing datasets, backed by Hadoop clusters. You install the Hadoop stack (localhost or distributed), then you can free-text search for datasets that have been made 'public' on any hadoop cluster that participates in the 'ecosystem'. We expect it to be self-policing, but there will be a way to report illegal distribution of datasets. The solution is based on…

That only addresses issue #3, which is much less important than issues #1 and #2. (and arguably worse: decentralization makes proper sourcing harder )

That's why we're working on a ranking/feedback system (a la piratebay) so that users can see other user comments on the datasets. Regarding decentralization - we are starting with centralized search, and download times are much better for popular datasets by downloading in parallel from many peers. We are also using a congestion control protocol (Ledbat) that is lower priority than TCP - you will not notice it using your b/w to share data if you are downloading/uploading using TCP.

Re: Show HN: Concept of a marketplace for machine learning datasets

#29

Post by jamesblonde »

Cacti wrote: post #27
jamesblonde wrote: post #17

Earlier quoted context omitted.

Good points. We are soon going to release a p2p system for sharing datasets, backed by Hadoop clusters. You install the Hadoop stack (localhost or distributed), then you can free-text search for datasets that have been made 'public' on any hadoop cluster that participates in the 'ecosystem'. We expect it to be self-policing, but there will be a way to report illegal distribution of datasets. The solution is based on…

Could you explain what problem you're trying to solve here? Are there really that many researchers who have access to modern (and expensive) GPU hardware that don't have bandwidth or disk space available? Or are there many researchers who are putting in lots of time assembling a dataset but don't have the bandwidth to distribute it?

It's more a case of providing a quick and easy way to share large datasets, backed by HDFS. So, researchers don't have a good way to share datasets (apart from AWS/GCE).

We work with climate science researchers who have multi-TB datasets, and they have no efficient way to share them. Same goes for genomics researchers who routinely pay lots of money for Aspera licenses just to download datasets faster than TCP allows. We are using a Ledbat protocol tuned to give good bandwidth over high latency links, but only scavange available b/w as it is lower priority than TCP.

For the machine learning researcher: i'd like to test this RNN on the reddit comments dataset....3 days later after finding a poor quality torrent...oh, now i can do it. On our system, search, find, click to download. We will move towards downloading (random) samples of very large datasets (even to Kafka from where they can be processed as they are downloaded).

Re: Show HN: Concept of a marketplace for machine learning datasets

#30

Post by pjackson5 »

Would there be much of an opportunity for someone whos into photo-realistic 3d rendering to create some of these datasets? For starters i was thinking of making something like the make3d dataset - http://make3d.cs.cornell.edu/data.html for some of my own experiments.
Post reply on HN