Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How ProPublica Illinois Uses GNU Make to Load Data

propublica.org

Post reply on HN
Print view View original

21–30 of 69 posts

Re: How ProPublica Illinois Uses GNU Make to Load Data

#21

Post by pdkl95 »

> [...] --ftp-passwd="$(ILCAMPAIGNCASH_FTP_PASSWD)" ftp://ftp.elections.il.gov/[...]

Is that using traditional (plaintext) FTP? Is it listening on port 21?

    ~ $ ftp ftp.elections.il.gov
    Connected to ftp.elections.il.gov (163.191.231.32).
    220-Microsoft FTP Service
    220 SBE
    Name (ftp.elections.il.gov): ^C
It looks like they are sending their password in plaintext. aria2 supports SFTP, so they should really talk to elections.il.gov about moving to SFTP or any other protocol that doesn't send the password in plaintext.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#22

Post by stakhanov »

drb91 wrote: post #17
stakhanov wrote: post #11

Was that supposed to say Petabytes? Gigabytes is really not that impressive.

It doesn’t seem like the size is supposed to be impressive, although I do not know why it is in the title. This is about the use of make.

...well, that was what I was trying to point out.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#24

Post by dang »

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#25

Post by danso »

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

As someone who has done a lot of data processing in journalism, I've found the engineering issues aren't usually about scale, but involve the harder problems around data cleaning/wrangling/updating. Particularly interoperability with opaque government systems, and transformation/delivery to a variety of users, including ones with a high variation in technical skill (i.e. journalists), and an extremely picky tolerance for public-facing errors in the finished product.

I started ProPublica's Dollars for Docs [0], and the initial project involved Today, the D4D has millions of records, and the government now its own website [1] for the official dissemination of the standardized data. I have a few shell scripts that can download the official raw data -- about ~30GB of text when unzipped -- and import it into a SQLite DB in about 20 minutes. The data for the first D4D investigation probably could've fit in a single Google Sheet, but it still took months to properly wrangle. But the computational bottleneck wasn't the size of data.

One of other tricky issues is that data management isn't easy in a newsroom. Devops is not only not a traditional priority, but anyone working with data has to do it fairly fast, and they have to move on almost immediately to another project/domain when done. There's not a lot of incentive or resources to get past a collection of hacky scripts, so it's really cool (to several-years-ago me) to see a guide about how to get things started in a more proper, maintainable way.

[0] https://projects.propublica.org/docdollars/

[1] https://www.cms.gov/openpayments/

edit: for a more technical detailed example of newsroom data issues, check out Adrian Holovaty's (creator of Django) 3-part essay, "Sane Data Updates Are Harder than You Think", which details the ETL process for Chicago crime data:

https://source.opennews.org/articles/sane-data-updates-are-h...

Here's a great write-up by Jeremy Merrill, who helped overhaul the D4D project after I left. Unlike me, Jeremy was a proper engineer:

https://www.propublica.org/nerds/heart-of-nerd-darkness-why-...

Re: How ProPublica Illinois Uses GNU Make to Load Data

#26

Post by TAForObvReasons »

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

Not just "open source technology", battle-tested tried-and-true technologies. Articles like these should remind us that we don't need to keep rebuilding tools to solve the same set of problems -- sometimes, all it takes is some familiarity with what exists

Re: How ProPublica Illinois Uses GNU Make to Load Data

#27

Post by WillEngler »

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

Hear hear! It's a good solution for this domain using reliable, free tools. And other folks without a lot of compute resources or experience setting up Spark clusters or whatever can easily adapt their approach. Hats off.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#28

Post by rainbowmverse »

TAForObvReasons wrote: post #26
rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

Not just "open source technology", battle-tested tried-and-true technologies. Articles like these should remind us that we don't need to keep rebuilding tools to solve the same set of problems -- sometimes, all it takes is some familiarity with what exists

I honestly didn't know Make did all that. I thought it was just a build script thing for complex software. This was enlightening.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#30

Post by bazizbaziz »

Minor nitpick about their exit code technique [0]: The command checks if the table exists, but it does not appear to re-run if the source file has been updated. Usually with Make you expect it to re-run the database load if the source file has changed.

It's better to use empty targets [1] to track when the file has last been loaded and re-run if the dependency has been changed.

[0] https://github.com/propublica/ilcampaigncash/blob/master/Mak...

[1] https://www.gnu.org/software/make/manual/html_node/Empty-Tar...

Post reply on HN