31–40 of 111 posts
Am I the only one who immediately thought of Drake the rapper? He's pretty famous, not sure if this was considered during the naming process. Even if it's not a legal problem, it's an SEO/social media problem.
Although I don't agree that the name "Drake" is an issue, I do find it interesting that an even more apt name for an application of this type might be "Usher"!
Am I the only one who immediately thought of Drake the rapper? He's pretty famous, not sure if this was considered during the naming process. Even if it's not a legal problem, it's an SEO/social media problem.
A drake is a male duck. They were pretty famous back in the day.
The initial system that I used was pretty similar to Paul Butler's technique, with a whole bunch of hacks to inform Make as to the status of various MySQL tables, and to allow jobs to be parallelized across the cluster.
At Custora, we needed a system specifically designed for running our various machine learning algorithms. We are always making improvements to our models, and we need to be able to do versioning to see how the improvements change our final predictions about customer behavior, and how these stack up to reality. So in addition to versioning code, and rerunning analysis when the code is out of date we also need to keep track of different major versions of the code, and figure out exactly what needs to be recomputed.
We did a survey of a number of different workflow management systems such as JUG, Taverna, and Kepler. We ended up finding a reasonable model in an old configuration management program called VESTA. We took the concepts from VESTA and wrote a system in Ruby and R to handle all of our workflow needs. The general concepts are pretty similar to to Drake, but it is specialized for our ruby and R modeling.
Some more useful links for those interested:
JUG https://github.com/luispedro/jug
Taverna http://www.taverna.org.uk/
Kepler https://kepler-project.org/
It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing ' The major differences I see are: - Inline support for Python et al. - Confirming the steps that will be taken. - HDFS support. Are there any other big differences?
I really wish that I had a tool like this back in grad school. I was doing bioinformatics work and merging, chopping, and processing various datasets over many months. When a new version of the underlying data came out it was not an easy task to go back and re-process it through dozens of steps in Perl and R. Having a tool like this would have made it a single command to do so and also ensured repeatability and trans…
As a fellow bioinformatician I can agree that this looks quite useful. Although (since you mention R), I wonder why there's no love for R in Drake, given that R is perhaps the quintessential data processing language.
Earlier quoted context omitted.
Please see my response to Make comparison: http://news.ycombinator.com/item?id=5111527 I suspect most of the points I made would be applicable to redo as well, if not more so. Trivial things don't require Drake. Heck, they often times don't require Make as well - just put it in a linear shell script if the steps are not too expensive. It's when things are getting complicated you need something like Drake.
Redo lacks features baked into Drake, especially the Hadoop integration, but I believe it would be easier to incorporate custom functionality into redo versus hacking Make or writing a custom build system. I haven't used Drake, so I would be interested in a small but complicated Drake script which tackles an intractable problem in Make. I don't claim redo can provide a cleaner solution than a purpose-built system, bu…
Other examples of intractable problems in Make would be timestamped dependency resolution between local and HDFS files. If Make can't look at HDFS, it can't say if the step needs to be built or not. I don't think you can fix it with external commands.
But generally, search for intractable problems is a futile one. Remember, everything you can code in Java, you can code in a Turing machine. :)
Am I the only one who immediately thought of Drake the rapper? He's pretty famous, not sure if this was considered during the naming process. Even if it's not a legal problem, it's an SEO/social media problem.
Djb redo[1], a make alternative, feels like a good fit for these type of data manipulation and dependency representations. Below is a port of the first example. The build script is just shell, so you can do stuff like embed python with a heredoc. One bit of syntactic sugar is that redo assumes stdout is the desired contents of the generated file, so you don't need to explicitly pipe to an OUTPUT variable. #!/bin/sh c…
The ideas behind redo are brilliant, but the way to express them in this particular implementation is not so fun.
I wrote a workflow processing system ( http://github.com/madhadron/bein ) that's still running around the bioinformatics community in southern Switzerland, and came to the conclusion that something like make isn't actually what you want. Unfortunately, what you want varies with the task at hand. The relevant parameters are: - The complexity of your analysis. - How fixed your pipeline is over time. - The size of a dat…
Why Clojure?