Introducing Drake, a kind of ‘make for data’
blog.factual.com
Introducing Drake, a kind of ‘make for data’
1–10 of 111 posts
Re: Introducing Drake, a kind of ‘make for data’
#2Why Clojure?
Re: Introducing Drake, a kind of ‘make for data’
#3Am I the only one who immediately thought of Drake the rapper? He's pretty famous, not sure if this was considered during the naming process. Even if it's not a legal problem, it's an SEO/social media problem.
Re: Introducing Drake, a kind of ‘make for data’
#4Why Clojure?
Probably because that's one of the languages that they use internally.
http://www.factual.com/jobs/oTR1Vfwq/Software-Engineer---Pal...
Another answer would be, "Why not?"
Re: Introducing Drake, a kind of ‘make for data’
#5I really wish that I had a tool like this back in grad school. I was doing bioinformatics work and merging, chopping, and processing various datasets over many months. When a new version of the underlying data came out it was not an easy task to go back and re-process it through dozens of steps in Perl and R. Having a tool like this would have made it a single command to do so and also ensured repeatability and transparency in my data, something which is often sorely lacking in an academic setting.
I am one of the data engineers at Factual and though I didn't have a role in creating it I definitely enjoy using it on a day to day basis. You begin to see the utility of it when you have a dozen people working up and down a data pipeline and need to coordinate as product specs evolve or schemas change.
I also really like the tagging features - you can add specific tags to different steps in the build and run different "flavors" of your workflow depending upon what is needed. For example, you might build a workflow that collects, cleans, filters, and performs calculations on data from all over the world - but you might also want alternative versions of the build that only work on specific regions or smaller debug datasets. Tags make that really simple to do, even when many steps are shared by the different versions or the dependencies are complicated.
Re: Introducing Drake, a kind of ‘make for data’
#6Am I the only one who immediately thought of Drake the rapper? He's pretty famous, not sure if this was considered during the naming process. Even if it's not a legal problem, it's an SEO/social media problem.
Although I don't agree that the name "Drake" is an issue, I do find it interesting that an even more apt name for an application of this type might be "Usher"!
Re: Introducing Drake, a kind of ‘make for data’
#7It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing 'The major differences I see are:
- Inline support for Python et al.
- Confirming the steps that will be taken.
- HDFS support.
Re: Introducing Drake, a kind of ‘make for data’
#8Djb redo[1], a make alternative, feels like a good fit for these type of data manipulation and dependency representations. Below is a port of the first example. The build script is just shell, so you can do stuff like embed python with a heredoc. One bit of syntactic sugar is that redo assumes stdout is the desired contents of the generated file, so you don't need to explicitly pipe to an OUTPUT variable.
#!/bin/sh
case $1 in
contracts.csv)
curl http://www.ferc.gov/docs-filing/eqr/soft-tools/sample-csv/contract.txt
;;
evergreens.csv)
redo-ifchange contracts.csv
grep Evergreen contracts.csv
;;
report.txt)
input=evergreens.csv
redo-ifchange $input
python2 Re: Introducing Drake, a kind of ‘make for data’
#9It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing ' The major differences I see are: - Inline support for Python et al. - Confirming the steps that will be taken. - HDFS support. Are there any other big differences?
Make was a major inspiration for us, and so Drake definitely has similarities to Make. The differences you list were non-trivial to us in usefulness, but of course YMMV. Also, there are a lot of (possibly) interesting future features described in the spec.
https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJ...
Re: Introducing Drake, a kind of ‘make for data’
#10It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing ' The major differences I see are: - Inline support for Python et al. - Confirming the steps that will be taken. - HDFS support. Are there any other big differences?
Make can support Python, or any other language you'd like. Just set ONESHELL to avoid splitting commands by line, and then set SHELL to your preferred language interpreter. Make will then hand that interpreter the entire body of commands to rebuild a target.