Was that supposed to say Petabytes? Gigabytes is really not that impressive.
How ProPublica Illinois Uses GNU Make to Load Data
11–20 of 69 posts
Re: How ProPublica Illinois Uses GNU Make to Load Data
#12[deleted]
Re: How ProPublica Illinois Uses GNU Make to Load Data
#13A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy.
If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporting once the government stopped updating its own tools.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#14A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…
Pettybytes. I like that.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#15Are you sure someone is not just streaming watching Netflix on your data plan? ... Honestly, I've seen Excel spread sheets load 10x more data than that! :D
Re: How ProPublica Illinois Uses GNU Make to Load Data
#16Original, https://www.propublica.org/nerds/gnu-make-illinois-campaign-...
I appreciate that it's under a /nerds path:)
Re: How ProPublica Illinois Uses GNU Make to Load Data
#17Was that supposed to say Petabytes? Gigabytes is really not that impressive.
It doesn’t seem like the size is supposed to be impressive, although I do not know why it is in the title. This is about the use of make.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#18Tbh I would have opted for Jenkins with declarative pipelines in his situation. Then you get logging , events , cron , CI , all more or less out of the box .
IMO, this is a bad use case for make.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#19Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me.
What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern.
Others have come up with different swiss army knives but nothing ever sticks for me, it usually ends up as a single Makefile with eg 3 targets that call a bunch of shell scripts.
The whole thing would be configurable to build from scratch, but not well set up to do incremental ETL on a per file basis, after I eg delete some extraneous rows in one file, clean up a column, redownload a folder, or add files to a dataset.
Re: How ProPublica Illinois Uses GNU Make to Load Data
#20A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporti…
Pettybytes. I like that.
[deleted]