Free, Public Data Sets • Hacker News Board

Re: Free, Public Data Sets

#21

Post by jmtame » Tue, Feb 01, 2011, 6:07 PM UTC

i've had trouble finding geographical boundaries on neighborhoods in U.S. cities (e.g. downtown areas and residential neighborhoods). anyone know where i can find this?

Re: Free, Public Data Sets

#22

Post by jamwt » Tue, Feb 01, 2011, 6:19 PM UTC

hvs wrote: post #17

Don't forget the Lahman Baseball Database with information from 1871-2010 http://baseball1.com/statistics/

And, for very detailed play-by-play data for decades of games, check out retrosheet: http://www.retrosheet.org/game.htm

Re: Free, Public Data Sets

#23

Post by _topher » Tue, Feb 01, 2011, 6:22 PM UTC

Thank you all for posting links and links to links to datasets, I have an unrelenting interest in data aggregation and machine learning, and didn't even know where to start. So helpful, and I am no longer stuck. :)

Re: Free, Public Data Sets

#24

Post by fedd » Tue, Feb 01, 2011, 6:38 PM UTC

do all of them have some uniformed api? that would be great, ideally. query and cache all of them on demand from your own app without additional programming.

bookmarked and shared this thread.

Re: Free, Public Data Sets

#25

Post by agbell » Tue, Feb 01, 2011, 7:02 PM UTC

Non-Free Google data:

http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=...

This data set, contributed by Google Inc., contains English word n-grams and their observed frequency counts.

Re: Free, Public Data Sets

#26

Post by kaffeinecoma » Tue, Feb 01, 2011, 7:17 PM UTC

This is a real treasure to come across. I hope we'll keep seeing jacquesm's blog postings here.

Anyone know of any publicly available song lyric databases?

Re: Free, Public Data Sets

#27

Post by bravura » Tue, Feb 01, 2011, 7:42 PM UTC

get.theinfo is the best way to find data sets. They are a bunch of data hoarders who can help you: http://groups.google.com/group/get-theinfo/?pli=1

I always ask there if I can't find what I'm looking for.

Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information.

    http://theinfo.org/
    http://infochimps.org/datasets
    http://ckan.org [Comprehensive Knowledge Archive Network]
    http://www.datawrangling.com/some-datasets-available-on-the-web.html
    http://del.icio.us/pskomoroch/dataset
    http://www.reddit.com/r/datasets/
    http://news.ycombinator.com/item?id=1242029
    http://www.reddit.com/r/opendata
    http://www.trustlet.org/wiki/Repositories_of_datasets
    http://www.daniel-lemire.com/blog/data-for-data-mining/
    http://www.quantlet.org/mdbase/
    http://datamob.org/
    http://freebase.com/
    http://infochimp.info/ics/data/ripd/www-personal.umich.edu/~mejn/netdata/
    http://www.archive-it.org/public/all_collections

    Large:
        http://www.ckan.net/tag/read/size-large
        http://www.diggingintodata.org/Repositories/tabid/167/Default.aspx

Web as corpus:

    Good instructions:
        http://corpus.leeds.ac.uk/internet.html#description
    http://sslmit.unibo.it/~baroni/bootcat.html

    http://www.drni.de/wac-tk/index.php/Documentation

etc. Email me if you need more http://cleaneval.sigwac.org.uk/ http://liste.sslmit.unibo.it/pipermail/sigwac/2007-November/... http://wacky.sslmit.unibo.it/doku.php?id= http://clic.cimec.unitn.it/marco/research.html