How ProPublica Illinois Uses GNU Make to Load Data • Hacker News Board

Re: How ProPublica Illinois Uses GNU Make to Load Data

#31

Post by jwilk » Wed, Jul 11, 2018, 7:46 PM UTC

pdkl95 wrote: post #21

> [...] --ftp-passwd="$(ILCAMPAIGNCASH_FTP_PASSWD)" ftp://ftp.elections.il.gov/[...] Is that using traditional (plaintext) FTP? Is it listening on port 21? ~ $ ftp ftp.elections.il.gov Connected to ftp.elections.il.gov (163.191.231.32). 220-Microsoft FTP Service 220 SBE Name (ftp.elections.il.gov): ^C It looks like they are sending their password in plaintext. aria2 supports SFTP, so they should really talk to electi…

This FTP server supports TLS:

  211-Extended features supported:                     
   LANG EN*
   UTF8
   AUTH TLS;TLS-C;SSL;TLS-P;
   PBSZ
   PROT C;P;
   CCC
   HOST
   SIZE
   MDTM
   REST STREAM

I didn't check if aria2 would actually use it, but I doubt it.

Re alternatives to FTP: as far as only downloads are concerned, HTTPS should be way easier to set up than SFTP.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#32

Post by outworlder » Wed, Jul 11, 2018, 7:48 PM UTC

dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

If you are creating files, and these files depends on other files, Make will do the dependency resolution for you. This is difficult to do with Bash and a huge reason to use Make.

Once you start abusing the ".PHONY" targets, the value starts to decrease.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#33

Post by duckerude » Wed, Jul 11, 2018, 7:49 PM UTC

dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

Make has a dependency system. You can tell it how to create file A, and that it depends on file B, and tell it how to create file B. Then if you request file A, it will check whether file B exists, and create it first if it's missing or outdated.

That's very valuable for building things. If you change a file, you only need to re-build the files that could reasonably be affected by the change. And things happen in the right order without micro-managing.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#34

Post by Pissompons » Wed, Jul 11, 2018, 7:51 PM UTC

dev_dull wrote: post #29

Can somehow help me understand the advantages of Make over a bash script? Isn’t bash superior in almost every way?

They can produce the same result but they do it in different ways and require you to express it in different ways.

Make has you describe a graph of outputs and how to produce them. It then traverses the graph to produce the requested output.

Bash is just a regular sequence of commands, with functions and loops if you wish.

If the pipeline you need to run can easily be turned into a dependency graph, I think make is a great fit. It's easy to use, comes with most of what you need built in and has some fun extras, like -jXXX, which allows you to parallelise things and built in caching so you don't regenerate the same asset twice if you don't need to.

You can do all that in bash but you'll have to write it yourself, which takes time you could spend on other things.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#35

Post by beebmam » Wed, Jul 11, 2018, 8:37 PM UTC

This is exactly the kind of purpose I love seeing open source tools used for. Kudos to propublica for leveraging open source to improve their ability to function!

Re: How ProPublica Illinois Uses GNU Make to Load Data

#36

Post by reacharavindh » Wed, Jul 11, 2018, 9:07 PM UTC

rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

Just today, I used xargs instead of spending a lot of time building a batching script in Python. I wanted to launch a bunch of processes in a queue but only execute 10 of them in parallel at any time.

Here is a skeleton of what I came up with.

    find $(pwd) -mindepth 1 -maxdepth 1 -type d -name ".zfs" -prune -o -type d -print0|xargs -0 -P 2 -I {} echo {}

where,

$(pwd) indicates the starting point of the listing of directories

-mindepth 1 makes sure current directory is not listed once again.

-maxdepth 1 makes sure the list does not get recursive

-type d -name - only directories and list names

".zfs" -prune - makes it ignore .zfs (snapshot directories)

-print0 - makes sure to print results without newlines. just -print will print one result per line

xargs -0 will take care of processing out spaces or newlines in the input stream

-P 2 — run two processes at once in parallel

-I {} says that replace {} in teh subsequent command from stdin piped into xargs echo {} will be echo dir1 and then echo dir2 etc

That's just an example to show that we can do a lot with standard Unix tools before bringing in the external sophistication for data related tasks.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#37

Post by danso » Wed, Jul 11, 2018, 9:16 PM UTC

rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

Someone on Twitter mentioned Luigi, which was previously developed and maintained by Spotify, as a distributed Make written with Python: https://github.com/spotify/luigi

Not sure if Spotify still uses it but it is in their Github org.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#38

Post by aslkdjaslkdj1 » Wed, Jul 11, 2018, 9:22 PM UTC

rockmeamedee wrote: post #19

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing e…

I'm a fan of Apache Airflow for large, complicated ETL processes especially those with depth and breadth in their dependencies.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#39

Post by lasermike026 » Wed, Jul 11, 2018, 9:25 PM UTC

I really like make! I use it almost every day. I like it for the structure and simplicity. I don't use it for everything. I plan to use it for the foreseeable future.

Why do I like make over shell scripting (sometimes) is that enforces structure. Shell scripts can turn into a real hairball.

When I did ruby I really enjoyed using rake.

Re: How ProPublica Illinois Uses GNU Make to Load Data

#40

Post by [deleted] » Wed, Jul 11, 2018, 9:29 PM UTC

dang wrote: post #24

rainbowmverse wrote: post #13

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…

Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.

[deleted]