Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Concept of a marketplace for machine learning datasets

datapie.in

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Show HN: Concept of a marketplace for machine learning datasets

#11

Post by akshaynathr »

nerdponx wrote: post #8
akshaynathr wrote: post #2

Hi everyone, While working on some of my projects involving Machine learning algorithms and deep neural networks i have found that there is a lack of training data sets in many areas. Also many of them are scattered throughout web, some are extremely huge for an individual to process etc.So i thought of this idea of having a marketplace for data ,structured for machine learning communities. It can be a one stop place…

Are you familiar with the UCI repository? https://archive.ics.uci.edu/ml/datasets.html

Yeah. I was thinking of a crowdsourced marketplace platform which follows a particular standard so all dataset supports major ML platforms, API support for programming languages for preprocessing huge datasets etc.People can prepare and sell /buy datasets. This can bring people from various streams together to solve really major issues using the data.

Re: Show HN: Concept of a marketplace for machine learning datasets

#13

Post by akshaynathr »

minimaxir wrote: post #7

There are a few services popping up with aim to provide data repositories for analysis/ML (Kaggle, data.world, /r/datasets) As someone who likes making analyses from random datasets, I have a few issues with these types of services: 1) There is often no indication of the distribution rights of the data, or whether the data was obtained ethically from the source (i.e. following the ToS). I made this mistake when I use…

Thanks for the valuable feedback.

Re: Show HN: Concept of a marketplace for machine learning datasets

#14

Post by hazelnut »

I think the idea is great but you should think about this sentence: "Buy and sell your data like Ebay" next to an image of connected people. It looks like you're a shady user profile dealer. To be successful it's crucial that you draw a clear line there

Re: Show HN: Concept of a marketplace for machine learning datasets

#15

Post by akshaynathr »

hazelnut wrote: post #14

I think the idea is great but you should think about this sentence: "Buy and sell your data like Ebay" next to an image of connected people. It looks like you're a shady user profile dealer. To be successful it's crucial that you draw a clear line there

I used that image to demonstrate the buyers and sellers being connected together. I will correct it asap.

Re: Show HN: Concept of a marketplace for machine learning datasets

#17

Post by jamesblonde »

minimaxir wrote: post #7

There are a few services popping up with aim to provide data repositories for analysis/ML (Kaggle, data.world, /r/datasets) As someone who likes making analyses from random datasets, I have a few issues with these types of services: 1) There is often no indication of the distribution rights of the data, or whether the data was obtained ethically from the source (i.e. following the ToS). I made this mistake when I use…

Good points. We are soon going to release a p2p system for sharing datasets, backed by Hadoop clusters. You install the Hadoop stack (localhost or distributed), then you can free-text search for datasets that have been made 'public' on any hadoop cluster that participates in the 'ecosystem'. We expect it to be self-policing, but there will be a way to report illegal distribution of datasets. The solution is based on a variant of Bittorrent where files are downloaded in-order (not randomly due to rarest piece selection in Bittorrent). Files can be downloaded to either HDFS or to a Kafka topic. We will demo it in 2 weeks here: https://fosdem.org/2017/schedule/event/democratizing_deep_le...

The system will be bootstrapped with lots of interesting big datasets: imagenet, 10m images, youtube 8m, reddit comments, hn comments, etc. Our experience is that we need a central point for researchers to get easy access to open datasets that doesn't require a AWS or GCE account.

Re: Show HN: Concept of a marketplace for machine learning datasets

#18

Post by verdverm »

hazelnut wrote: post #14

I think the idea is great but you should think about this sentence: "Buy and sell your data like Ebay" next to an image of connected people. It looks like you're a shady user profile dealer. To be successful it's crucial that you draw a clear line there

also, how does this fit with...

""" Datapie offers data analysis without downloading the data. This means you need not download the massive data. No need to have massive distributed systems to process it. """

Where is your company trying to fit into the market? Is this a http://zerotoonebook.com/ or are we commodotized in this space already?

IBM offers many of the same data sets, paired with your company's private data, both annotated by Watson. IBM also does not learn from, or improve their own models, w.r.t. your company's proprietary data.

Re: Show HN: Concept of a marketplace for machine learning datasets

#20

Post by huac »

akshaynathr wrote: post #11
nerdponx wrote: post #8

Earlier quoted context omitted.

Are you familiar with the UCI repository? https://archive.ics.uci.edu/ml/datasets.html

Yeah. I was thinking of a crowdsourced marketplace platform which follows a particular standard so all dataset supports major ML platforms, API support for programming languages for preprocessing huge datasets etc.People can prepare and sell /buy datasets. This can bring people from various streams together to solve really major issues using the data.

what is a 'particular standard for data'
Post reply on HN