FiveThirtyEight has a GitHub repo with story-related data and scripts

This is great. Next up, academic researchers please...

Sorry to seemingly play a game of one-upsmanship, as 538's repo is wonderful, but BuzzFeed also posts its data and stories online on their BuzzFeedNews Github account...however, what I really like about their repo setup is that they have a repo for every project...and then a separate repo that has a tabular listing of repo, date, description, and story link * ...IMO, they have the best setup (in terms of discovery) among the news nerds:

https://github.com/BuzzFeedNews/everything

- * edit: Oops, I forgot that 538's data repo also has a readable table listing below the fold...I suppose having separate repos per dataset vs one main repo is not a clear advantage depending on the user.

They also list their standalone datasets and tools, e.g. their standardized H-2 certification data [1], for which they've probably used in their many H-2 visa stories, and twick [2], the tool they use to quickly fetch newsworthy Twitter account data on short notice...

And those aren't even all the useful tools that that team has created...Jeremy Singer-Vine (their data editor) has several great Python utilities in his personal repo (github.com/jsvine), including waybackpack, pdfplumber (a wrapper around pdfminer and similar to Ruby's tabula), and markovify.

[1] https://github.com/BuzzFeedNews/H-2-certification-data

[2] https://github.com/jsvine/twick

I'm reading through Nate Silver's Signal vs. Noise book right now - highly recommend it. And this repo is an awesome way to replicate the results that they show on their site (and perhaps some stuff in the book too).

The importance of statistical transparency is part of the reason why I've switched to GitHub and Jupyter notebooks, as GitHub renders them natively which makes it obvious what code is being run and what the results of the code are. (two recent examples of mine are processing AngelList data [https://github.com/minimaxir/sfba-compensation/blob/master/a...] and creating graph networks from Reddit data [https://github.com/minimaxir/reddit-graph/blob/master/subred... ])

That said, there are a few reasons I've seen writers/researchers intentionally not release data. Either the code is sloppy/inefficient/embarrasing, or the data is used in bad content marketing for a startup that specializes in data collection so that they can say "we have data, pay us if you want more, neener neener neener." (The latter of which I see submitted to HN all the time and serves as a pet peeve of mine)

I mean kinda...

http://fivethirtyeight.com/features/how-i-acted-like-a-pundi...

http://fivethirtyeight.com/features/elections-podcast-paul-r...