How ProPublica Illinois Uses GNU Make to Load Data • Hacker News Board

Re: How ProPublica Illinois Uses GNU Make to Load Data

#61

Post by reacharavindh » Thu, Jul 12, 2018, 11:52 AM UTC

icebraining wrote: post #59

reacharavindh wrote: post #36

Earlier quoted context omitted.

Just today, I used xargs instead of spending a lot of time building a batching script in Python. I wanted to launch a bunch of processes in a queue but only execute 10 of them in parallel at any time. Here is a skeleton of what I came up with. find $(pwd) -mindepth 1 -maxdepth 1 -type d -name ".zfs" -prune -o -type d -print0|xargs -0 -P 2 -I {} echo {} where, $(pwd) indicates the starting point of the listing of dire…

And with GNU parallel, which can take the place of xargs, you can even distribute that job across multiple machines easily (as long as they're accessible by SSH).

Yes, I need to look into whether and how Gnu Parallel will queue up tasks if I restricted the number of parallel processes.

In my case, I was dealing with a FreeBSD server. I went the xargs route instead of installing something that is not available by default.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#62

Post by catacombs » Thu, Jul 12, 2018, 2:31 PM UTC

beebmam wrote: post #35

This is exactly the kind of purpose I love seeing open source tools used for. Kudos to propublica for leveraging open source to improve their ability to function!

While it's nice to see Propublica use open source software, keep in mind dozens and dozens of other news organizations use the same tools.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#63

Post by catacombs » Thu, Jul 12, 2018, 2:32 PM UTC

yjftsjthsd-h wrote: post #16

brudgers wrote: post #2

Original, https://www.propublica.org/nerds/gnu-make-illinois-campaign-...

I appreciate that it's under a /nerds path:)

It's cute.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#64

Post by mmt » Thu, Jul 12, 2018, 3:55 PM UTC

pletnes wrote: post #60

mmt wrote: post #53

Earlier quoted context omitted.

In addition to all the features the sibling comments noted, it's important to note that there's no contradiction: bash is usually the scripting language one uses inside of a Makefile. It's the default, although one could use any scripting language. Point being, there's no "Make" language, beyond the syntax for describing those dependency relationships and variable assignments.

I believe /bin/sh is the default, not bash. But this can be changed.

You're right that it's /bin/sh, but, since it could be (and is, in some cases) bash, it's not quite right to call it "not bash", either.

I'll grant that the distinction is important, though, in the face of the history of #!/bin/sh Linux scripts with bashisms breaking upon the Debian/Ubuntu switch to dash. Even if you're on a system where /bin/sh is bash, it's safest to set SHELL in your GNU makefiles to bash explicitly, if that's what you you're writing in.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#65

Post by stakhanov » Thu, Jul 12, 2018, 6:47 PM UTC

On the whole debate revolving around gigabytes in the title, I'd like to add:

There's a well-substantiated linguistic theory revolving around "maxims of conversation". Maxims of conversation are so strongly universal among the speakers of a given language that they become part of the implied meaning of a conversational act.

For example the maxim of cooperativitiy implies that when a person sitting in a cold room next to a window is spoken to by a person sitting further from the window and is being told "It's a bit chilly, isn't it", they can take it to mean "Please close the window".

https://en.wikipedia.org/wiki/Implicature#Conversational_imp...

Similarly, there are certain maxims of conversation which are part of the language game inherent in the formulation of the title of a blogpost. They are kind of assumed to be boasting about something. So when somebody says "We figured out a way to load a gigabyte's worth of data into a database in a single day" then the being-boastful-about-something maxim is violated. That's why it triggered so many people.

And pointing out that this is not something to be boastful about is a perfectly valid thing to do to keep certain facts straight.

...just saying.

But, by all means, if you get a thrill out of it, keep downvoting me.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#66

Post by perl4ever » Thu, Jul 12, 2018, 7:37 PM UTC

mmt wrote: post #54

danso wrote: post #25

Earlier quoted context omitted.

As someone who has done a lot of data processing in journalism, I've found the engineering issues aren't usually about scale, but involve the harder problems around data cleaning/wrangling/updating. Particularly interoperability with opaque government systems, and transformation/delivery to a variety of users, including ones with a high variation in technical skill (i.e. journalists), and an extremely picky tolerance…

I suspect that anyone who has worked in tech at more "traditional" non-tech businesses would be far more familiar with the challenges inherent in any ETL undertaking. It's usually critical business data, too, so there's a strong incentive to avoid errors there, too. The trouble is, despite (or possibly because of) being cognitively difficult and requiring a certain discipline (for lack of a better word), this kind of…

It isn't even about tech vs "non-tech". It's about whether you get data in a consistent format or not. Where I used to work, we would get a gigabyte sized file of random XML without any documentation and be told to deal with it, first step being to tell the non technical people what we had. Another delivery would be something totally different. Saying "oh, we deal with petabytes" is missing the point. If there's nothing unexpected or unknown about the data, then it's not a challenge, because you know, computers process stuff automatically.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#67

Post by nineteen999 » Thu, Jul 12, 2018, 11:47 PM UTC

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

The production databases on the project I build and support around are around 100GB in size - tiny. But if they didn't work correctly - ambulances wouldn't arrive at the correct location on time, nor fire engines nor emergency services workers.

Covering an area of 1.25 million square kilometres, supporting 40,000 first responders helping to protect 8 million people.

Of course the databases are not the only important part of an emergency services network such as ours, but they are a critical component.

I would rather work on a project like this any day than working to prop up some faceless advertising/data collection behemoth such as Facebook or Google.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#68

Post by mmt » Fri, Jul 13, 2018, 3:53 AM UTC

perl4ever wrote: post #66

mmt wrote: post #54

Earlier quoted context omitted.

I suspect that anyone who has worked in tech at more "traditional" non-tech businesses would be far more familiar with the challenges inherent in any ETL undertaking. It's usually critical business data, too, so there's a strong incentive to avoid errors there, too. The trouble is, despite (or possibly because of) being cognitively difficult and requiring a certain discipline (for lack of a better word), this kind of…

It isn't even about tech vs "non-tech". It's about whether you get data in a consistent format or not. Where I used to work, we would get a gigabyte sized file of random XML without any documentation and be told to deal with it, first step being to tell the non technical people what we had. Another delivery would be something totally different. Saying "oh, we deal with petabytes" is missing the point. If there's noth…

I guess I was making the assumption that "tech" businesses are more likely to have data that's entirely generated by modern software (e.g. click logs) or at least pre-coerced into a consistent, if not structured format (e.g. tweets), whereas non-tech businesses are more likely to have data that's free-form human input or comes disparate/arbitrary machine sources (e.g. scientific instruments, the mainframe or AS/400 worlds).

I'm sure there's a spectrum, but my point was that the vast majority of what the companies we read about on this site ("tech") deal with is going to fall close to the consistent-format edge of the spectrum, hence the prejudice.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#69

Post by dang » Fri, Jul 13, 2018, 4:59 AM UTC

virtuabhi wrote: post #42

dang wrote: post #24

Earlier quoted context omitted.

Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.

Would it be possible to put the gigabytes back in the title? In general, ETL of gigabytes of data can involve complicated operations, e.g., use of statistical models. And, the utility of data is not determined by the size of data. One has to be a pretty petty person to make fun of this article for data sizes.

You're right on the facts, but that's not the way a forum like this works. Like it or not (and probably no one likes it), the way it works is that a minor irrelevant provocation in the title completely determines the discussion. The solution is to take out the minor provocation, even if in principle we shouldn't need to.