11–20 of 111 posts
I really like the inline, multi-language scripting though.
It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing ' The major differences I see are: - Inline support for Python et al. - Confirming the steps that will be taken. - HDFS support. Are there any other big differences?
The concept of Make is not unique. Everything that has dependencies and executes steps is similar to Make in concept. Drake is no exception, and it can be replaced with Make, but no more so than Rake, Ant or Maven can be replaced by Make. That is, if it's trivial - yes. Just a bit more complicated - no.
Some things are merely painful to implement with Make, some are just impossible:
- multiple outputs
- no-input and no-output steps
- HDFS support
- Hadoop's partial files support (part-?????)
- forced execution of any subbranch, up or down the tree or any individual targets (crucial for debugging and development)
- target exclusions
- protocol abstraction - inline Python is just one example
- tags
- branching
- methods
- automated data versioning (backup and revert)
- parallelization
- real-time status console
- retries, email notifications
- etc.
I guess, if Make was really, really extendable, we could have considered it as a platform for all this. But it's not, and hacking all of that into Make's source code in C would be, I'm sure, a much greater pain than writing Drake.
Artem.
It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing ' The major differences I see are: - Inline support for Python et al. - Confirming the steps that will be taken. - HDFS support. Are there any other big differences?
Make can support Python, or any other language you'd like. Just set ONESHELL to avoid splitting commands by line, and then set SHELL to your preferred language interpreter. Make will then hand that interpreter the entire body of commands to rebuild a target.
I see that Drake is implemented in Clojure, so I'd imagine you understand the value of homoiconicity and extensible languages. So I wonder why you didn't just use Clojure all the way through?
Djb redo[1], a make alternative, feels like a good fit for these type of data manipulation and dependency representations. Below is a port of the first example. The build script is just shell, so you can do stuff like embed python with a heredoc. One bit of syntactic sugar is that redo assumes stdout is the desired contents of the generated file, so you don't need to explicitly pipe to an OUTPUT variable. #!/bin/sh c…
http://news.ycombinator.com/item?id=5111527
I suspect most of the points I made would be applicable to redo as well, if not more so. Trivial things don't require Drake. Heck, they often times don't require Make as well - just put it in a linear shell script if the steps are not too expensive. It's when things are getting complicated you need something like Drake.
I like the idea that the tasks can be implemented in any language, but I feel like this has limitations compared to something like Rake, where the step definition is code, too. What this means is that in Rake I am not just limited to defining new task bodies, but new ways of defining tasks themselves. I see that Drake is implemented in Clojure, so I'd imagine you understand the value of homoiconicity and extensible l…
http://www.youtube.com/watch?feature=player_detailpage&v...
In short, we don't feel like it's an either or question. We want to have Drake as a command-line frontend to the core functionality, but we would love to see/have other frontends developed as well. Currently, there's no Clojure DSL for Drake, but I think it'd be totally awesome.
The reason we started from command-line is because our workflows are heterogenous, and we also didn't want to limit Drake to developers and associate it with coding. Clojure can be quite a big learning curve if you only need it to specify steps and link them together through file dependencies.
We had an important design goal in mind: Drake should be as simple as writing a shell script. If it's not, our experience shows that most workflow start as trivial shell-scripts with one or two steps, and by the time it grows into something unmanageable, it's kinda too late. :)
On a related note, Drake supports Clojure code inlining for manipulation of the parse tree. It's not an equivalent, just a somewhat related feature. It allows you to modify the steps, dependencies, and anything else in the parse tree directly from Clojure.
I like the idea that the tasks can be implemented in any language, but I feel like this has limitations compared to something like Rake, where the step definition is code, too. What this means is that in Rake I am not just limited to defining new task bodies, but new ways of defining tasks themselves. I see that Drake is implemented in Clojure, so I'd imagine you understand the value of homoiconicity and extensible l…
There seems to be few differences between Drake and just rolling out Makefiles for data processing, but I definitely see this project has potential. Distributed processing over AWS/Compute Engine/etc. clusters would be one nice thing to have, as a kind of simpler alternative to Hadoop. I really like the inline, multi-language scripting though.
Reminds me of Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids, Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET) at ACM SIGMOD, May, 2012. https://www3.nd.edu/~ccl/software/makeflow/