Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Datasets for Machine Learning

gengo.ai

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Datasets for Machine Learning

#31

Post by benhamner »

bhnmmhmd wrote: post #12
benhamner wrote: post #7

Ben from Kaggle. Open up the ~50 different individual datasets linked in separate tabs, and then quickly flip through all of them trying to get a sense of what each one is. That experience will demonstrate one of the main challenges we're aiming to solve by making Kaggle Datasets your default place to publish data online ( https://www.kaggle.com/datasets )

I've heard that Kaggle data sets encourage people to do "supervised" ML only. Is that true?

The competitions we host (https://www.kaggle.com/competitions) are supervised and always have a target we can create a numeric leaderboard on, but the public datasets (https://www.kaggle.com/datasets) are used for everything under the sun.

There's some supervised ML use of those, and a lot more open-ended exploration, visualization, cleaning, clustering, language modeling, etc.

Re: Datasets for Machine Learning

#32

Post by benhamner »

benhamner wrote: post #7

Ben from Kaggle. Open up the ~50 different individual datasets linked in separate tabs, and then quickly flip through all of them trying to get a sense of what each one is. That experience will demonstrate one of the main challenges we're aiming to solve by making Kaggle Datasets your default place to publish data online ( https://www.kaggle.com/datasets )

You can also start a new Jupyter notebook session on any of these datasets with a click (click "New Kernel"), and then accelerate your analysis by attaching a GPU to the session with another click (for applications a GPU helps, e.g. training Tensorflow models on image data)

Re: Datasets for Machine Learning

#34

Post by danso »

Two sources that are missing:

opendatanetwork.com: this is effectively a Google for public Socrata data portals, and for me, the best way to discover datasets across different municipalities. For example, when I was interested in trying to replicate the NYT's "Do ‘Fast and Furious’ Movies Cause a Rise in Speeding?" [0] article, it was pretty easy to find a bunch of other traffic/motor vehicle violation datasets with opendatanetwork's search.

Enigma public (https://public.enigma.com): a huge collection of scraped public datasets, including flattened versions of data that originally comes in annoying-to-parse, such as U.S. lobbying disclosures [1]

[0] https://www.nytimes.com/2018/01/30/upshot/do-fast-and-furiou...

[1] https://public.enigma.com/datasets/lobbying-disclosures-lobb...

Re: Datasets for Machine Learning

#36

Post by TuringNYC »

benhamner wrote: post #27
fursund wrote: post #15

Earlier quoted context omitted.

This is great! Thanks for sharing. Would be awesome if your license filter had a "not for commercial use" vs. "for commercial use" or similar.

Thanks for the feedback! Totally agree this could be clearer

Same question as parent...but curious...which one is it?

Re: Datasets for Machine Learning

#39

Post by akarve »

benhamner wrote: post #7

Ben from Kaggle. Open up the ~50 different individual datasets linked in separate tabs, and then quickly flip through all of them trying to get a sense of what each one is. That experience will demonstrate one of the main challenges we're aiming to solve by making Kaggle Datasets your default place to publish data online ( https://www.kaggle.com/datasets )

Ironically, the same challenges are better solved in the world of code: Docker, GitHub, npm, etc.

Some friends and I created Quilt to bring versioning and packaging to data: https://quiltdata.com/. The interface is the familiar Python lifecycle of install and import.

Post reply on HN