Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How ProPublica Illinois Uses GNU Make to Load Data

propublica.org

Post reply on HN
Print view View original

51–60 of 69 posts

Re: How ProPublica Illinois Uses GNU Make to Load Data

#51

Post by rspeer »

rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

I use Snakemake [1], a parallel make system for data, designed around pattern-matching rules. The rules are either shell commands or Python 3 code.

I settled on it after originally using make, getting frustrated with the crazy work-arounds I needed to implement because it doesn't understand build steps with multiple outputs, switching to Ninja where you have to construct the dependency tree yourself, and finally ending up on Snakemake which does everything I need.

[1] https://snakemake.readthedocs.io/en/stable/

Re: How ProPublica Illinois Uses GNU Make to Load Data

#52

Post by tejtm »

dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

Make is an excellent way to automatically decide to run your bash scripts (and other scripts, shell commands & executables) or not, all depending on if the existing output is newer than the available input

Re: How ProPublica Illinois Uses GNU Make to Load Data

#53

Post by mmt »

dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

In addition to all the features the sibling comments noted, it's important to note that there's no contradiction:

bash is usually the scripting language one uses inside of a Makefile.

It's the default, although one could use any scripting language. Point being, there's no "Make" language, beyond the syntax for describing those dependency relationships and variable assignments.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#54

Post by mmt »

danso wrote: post #25
rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

As someone who has done a lot of data processing in journalism, I've found the engineering issues aren't usually about scale, but involve the harder problems around data cleaning/wrangling/updating. Particularly interoperability with opaque government systems, and transformation/delivery to a variety of users, including ones with a high variation in technical skill (i.e. journalists), and an extremely picky tolerance…

I suspect that anyone who has worked in tech at more "traditional" non-tech businesses would be far more familiar with the challenges inherent in any ETL undertaking. It's usually critical business data, too, so there's a strong incentive to avoid errors there, too.

The trouble is, despite (or possibly because of) being cognitively difficult and requiring a certain discipline (for lack of a better word), this kind of work doesn't come across as very "sexy" anecdotally.

Even if it does get shared, the part that makes it hard gets overlooked.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#55

Post by mturmon »

dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

The sibling comments give the top-level answer.

In addition, expanding on @Pissompons's note -- make gives you job-level parallelism for free with constructs like:

    make -j 24 transform
which will (if possible/allowed by the dependence structure in the Makefile) run 24 jobs at once to bring "transform" up to date.

So for instance, if "transform" depends on a bunch of targets, one for each month across a decade, you get 24-way parallelism for free. It's kind of like gnu "xargs -P", but embedded within the make job-dispatcher.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#56

Post by therein »

flukus wrote: post #49
rasz wrote: post #5

Earlier quoted context omitted.

aria is multi connection(aria2c -x5 means five concurrent), thats the main reason for speed bump

Does this increase speed when it's only downloading a single file at a time? It might be better off using makes multi process (make -j 5) to be able to process data while still loading other data.

Each connection requests a different range within the same file and they download together.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#57

Post by reacharavindh »

rspeer wrote: post #51
rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

I use Snakemake [1], a parallel make system for data, designed around pattern-matching rules. The rules are either shell commands or Python 3 code. I settled on it after originally using make, getting frustrated with the crazy work-arounds I needed to implement because it doesn't understand build steps with multiple outputs, switching to Ninja where you have to construct the dependency tree yourself, and finally endi…

Thank you for sharing this information about snakemake. I administer a cluster for a group of geneticists. I'll try to get them to use it for their publications to make their results easily reproducible by others.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#58

Post by IanCal »

danso wrote: post #37
rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

Someone on Twitter mentioned Luigi, which was previously developed and maintained by Spotify, as a distributed Make written with Python: https://github.com/spotify/luigi Not sure if Spotify still uses it but it is in their Github org.

Luigi is great although I don't think it's easy to add "rerun if source file updated". Would love to be wrong on that.

http://pachyderm.io seems great but does require more engineering support (needs a kubernetes cluster)

Re: How ProPublica Illinois Uses GNU Make to Load Data

#59

Post by icebraining »

reacharavindh wrote: post #36
rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

Just today, I used xargs instead of spending a lot of time building a batching script in Python. I wanted to launch a bunch of processes in a queue but only execute 10 of them in parallel at any time. Here is a skeleton of what I came up with. find $(pwd) -mindepth 1 -maxdepth 1 -type d -name ".zfs" -prune -o -type d -print0|xargs -0 -P 2 -I {} echo {} where, $(pwd) indicates the starting point of the listing of dire…

And with GNU parallel, which can take the place of xargs, you can even distribute that job across multiple machines easily (as long as they're accessible by SSH).

Re: How ProPublica Illinois Uses GNU Make to Load Data

#60

Post by pletnes »

mmt wrote: post #53
dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

In addition to all the features the sibling comments noted, it's important to note that there's no contradiction: bash is usually the scripting language one uses inside of a Makefile. It's the default, although one could use any scripting language. Point being, there's no "Make" language, beyond the syntax for describing those dependency relationships and variable assignments.

I believe /bin/sh is the default, not bash. But this can be changed.
Post reply on HN