Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How ProPublica Illinois Uses GNU Make to Load Data

propublica.org

Post reply on HN
Print view View original

41–50 of 69 posts

Re: How ProPublica Illinois Uses GNU Make to Load Data

#41

Post by danso »

pdkl95 wrote: post #21

> [...] --ftp-passwd="$(ILCAMPAIGNCASH_FTP_PASSWD)" ftp://ftp.elections.il.gov/[...] Is that using traditional (plaintext) FTP? Is it listening on port 21? ~ $ ftp ftp.elections.il.gov Connected to ftp.elections.il.gov (163.191.231.32). 220-Microsoft FTP Service 220 SBE Name (ftp.elections.il.gov): ^C It looks like they are sending their password in plaintext. aria2 supports SFTP, so they should really talk to electi…

I imagine there would be other systems (state-owned and private) that use the FTP server, and maybe in a way that changing protocols is inexplicably full of friction. I wonder why the elections server, assuming it only contains records legal to distribute to the public, is even password protected. Maybe it was a policy when govt bandwidth was scarce. California, for example, has campaign finance data on a public webserver: https://www.californiacivicdata.org/

And the FEC has an API, but has long had the data hosted on public FTP: https://classic.fec.gov/finance/disclosure/ftp_download.shtm...

Re: How ProPublica Illinois Uses GNU Make to Load Data

#42

Post by virtuabhi »

dang wrote: post #24
rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.

Would it be possible to put the gigabytes back in the title? In general, ETL of gigabytes of data can involve complicated operations, e.g., use of statistical models. And, the utility of data is not determined by the size of data. One has to be a pretty petty person to make fun of this article for data sizes.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#43

Post by chemicalcrux »

I did something like this a few years ago! I needed to do a bunch of transformations and measurements of data that came in on a regular basis. Make was a perfect fit - I could test the whole process with a single command, cleaning either just the result data, or nuking everything to make sure it pulled stuff in properly.

I spent some time trying to write my own processing system in Python before realizing this was a familiar task...

Re: How ProPublica Illinois Uses GNU Make to Load Data

#44

Post by qubax »

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

> The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads.

No it isn't. You are getting defensive for no reason. If propublica ceases to exist or never existed, it wouldn't matter a single bit to the world. You could even argue the world would be better off.

> Processing petty petabytes is not praiseworthy.

From a technical point and many other ways, it is.

I don't get why you are getting offended by people making a jab at the scant amount of data. Last I checked, hacker news is a technology oriented site. And from a technology point of view, what pro publica is doing is a joke. It's a toy amount of data.

Why not just say pro publica is not a technology company and hence people shouldn't expect technological feats of wonder?

> The point is they used open source technology to process public data for reporting once the government stopped updating its own tools.

Which is something I could have done on a lazy afternoon all by myself. It isn't anything to be impressed about. But good for them anyways.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#45

Post by LMYahooTFY »

qubax wrote: post #44
rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

> The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. No it isn't. You are getting defensive for no reason. If propublica ceases to exist or never existed, it wouldn't matter a single bit to the world. You could even argue the world would be better off. > Processing petty petabytes is not praiseworthy. From a technical point and many other ways,…

So people can't make a point about any aspects of accomplishment that are not engineering feats?

Whether or not Propublica has produced something of value for society seems, at an absolute minimum, highly debatable.

Your comment is the one that seems defensive....

Re: How ProPublica Illinois Uses GNU Make to Load Data

#46

Post by mc92nxk »

qubax wrote: post #44
rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

> The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. No it isn't. You are getting defensive for no reason. If propublica ceases to exist or never existed, it wouldn't matter a single bit to the world. You could even argue the world would be better off. > Processing petty petabytes is not praiseworthy. From a technical point and many other ways,…

The way qubax sees the world is as important as qubax makes propublica’s view out to be

You’re being too literal. Yes if nothing exists it doesn’t matter

Look in the mirror and realize you’re just one of thousands that could do this in an afternoon

Given the “big picture” context, your personal computer skills aren’t much to brag about either. Literally good with computers. Get in line.

Were they being defensive? Or offering a context to consider the value from?

Re: How ProPublica Illinois Uses GNU Make to Load Data

#47

Post by danso »

qubax wrote: post #44
rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

> The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. No it isn't. You are getting defensive for no reason. If propublica ceases to exist or never existed, it wouldn't matter a single bit to the world. You could even argue the world would be better off. > Processing petty petabytes is not praiseworthy. From a technical point and many other ways,…

ProPublica is not a technology company, it's a non-profit investigative journalism outlet.

They and countless other journalism/civic orgs would likely be happy for you to show them up by whipping up usable ETL scripts relevant in their respective domains. Since it all involves public open data you don't have to wait for anyone's permission.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#48

Post by rainbowmverse »

virtuabhi wrote: post #42
dang wrote: post #24

Earlier quoted context omitted.

Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.

Would it be possible to put the gigabytes back in the title? In general, ETL of gigabytes of data can involve complicated operations, e.g., use of statistical models. And, the utility of data is not determined by the size of data. One has to be a pretty petty person to make fun of this article for data sizes.

That was almost every comment before I posted. All of them seem to be grayed out now.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#49

Post by flukus »

rasz wrote: post #5
danso wrote: post #3

> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes. Tangential…

aria is multi connection(aria2c -x5 means five concurrent), thats the main reason for speed bump

Does this increase speed when it's only downloading a single file at a time? It might be better off using makes multi process (make -j 5) to be able to process data while still loading other data.
Post reply on HN