Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Drake, a kind of ‘make for data’

blog.factual.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Introducing Drake, a kind of ‘make for data’

#71

Post by aboytsov »

danpalmer wrote: post #68
aboytsov wrote: post #62

Earlier quoted context omitted.

Hey, thanks for trying out our tool! First of all, --version shouldn't try to run any targets. This seems like a bug. Thanks. Yes, you guessed correctly - this is the JVM startup time. I just hate JVM for that. We experimented with Nailgun and Drip to eliminate it - Nailgun is problematic because it uses a shared JVM for all runs, and it can get quite hairy sometimes. In the long run, Nailgun is almost certainly not…

Thanks for the detailed and well explained reply. I have limited experience with Clojure, but it does seem to be a good match to this sort of task due to it's structure. However the JVM seems to be a real drawback to me. Perhaps with something like Scheme or Lisp you might get a similar program structure, and be able to compile to faster binaries? The REPL is a solution, but as many developers are using tools like ma…

I can certainly see your point about using Drake in an automated environment where this delay would still matter, but running a daemon is not practical. I think you have a lot of good arguments against JVM. There were some moments when I thought it might not have been the best choice as well - for example, Java world is notoriously poor with dealing with child processes.

So, I agree, but there are several arguments that it's not that bad after all:

- Drake is fundamentally an interactive tool. If you run it as a part of an automated process, all its flexibility is not quite needed. You could have Drake print a list of all shell commands it would execute, and save it to get your automated script.

- Most data workflows Drake is good for are quite expensive. Minutes, sometimes hours. Definitely much more than 5 seconds. The reason is simple - if your workflow takes so little time, you're really not gaining much by using a complicated tool like Drake, instead of just putting it all in a linear shell script, and simply re-running everything every time you need it.

- Maybe we'll find a good solution like Nailgun and Drip.

- Maybe someone will make a Java-code compiler that would create a stand-alone executable out of a JAR.

- Maybe Sun will eliminate JVM startup overhead. Or somebody will release a 3rd party JVM without it.

- Maybe we'll have a compiled version of Clojure one day.

- Other maybes. :)

We certainly would support any effort to port Drake into Lisp, C++, Ruby, Python or any language you desire. Porting it into Common Lisp might not be that much easier than to Ruby. We might not consider it ourselves, since the effort will be quite substantial.

Does it sound reasonable to you?

Re: Introducing Drake, a kind of ‘make for data’

#72

Post by tedunangst »

aboytsov wrote: post #66
tedunangst wrote: post #58

Earlier quoted context omitted.

This is a really weird request. make won't rebuild things that haven't changed, so the default make all rule will only rebuild the things depending on mytarget. Every time you change mytarget, just run make (all) and everything that depends on mytarget (and only those things) will be rebuilt.

Really? This is not a weird request, this is one of the most common things we do when we're developing a workflow. You need to do this every time you make changes to code and you want these changes to propagate. You can't run "make all", because it literally builds everything. You might be working on a specific branch of the workflow, and the overall workflow could be huge. And out-of-date in a lot of places. Or it c…

Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.

Re: Introducing Drake, a kind of ‘make for data’

#73

Post by zmmmmm »

aboytsov wrote: post #69
zmmmmm wrote: post #64

Earlier quoted context omitted.

Heh, all the bioinformaticians come out of the woodwork :-) Here's yet-another-project for bioinformatics workflows that I've been involved in. This one based on Groovy: http://bpipe.org I agree with your sentiments about the nature of pipelines vs build system a la make. Many many people start down the path of putting the classic DAG dependency analysis as the foundation of their needs when in fact, this isn't so mu…

Thanks for your feedback. We do mention parallelization in the designdoc, it's just not implemented yet. It's quite easy to add though. We have a lot of features spec'ed out, but not implemented. I would appreciate if you elaborated on separating step definitions from dependency definitions. In my mind, they are the same thing. If you mean that steps might not be connected by input-output relationship, but still have…

> I would appreciate if you elaborated on separating step definitions from dependency definitions

As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page:

    evergreens.csv  $OUTPUT
So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now

    renamed.csv  $OUTPUT

    evergreens.csv  $OUTPUT
Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say

    extract_evergreens = { 
      exec "grep Evergreen $input > $output" 
    }

    fix_names = { 
      exec "sed 's/Neverbrown/Evergreen/g' $input > $output"
    }
Then you define your pipeline order separately -

    run { fix_names + extract_evergreens }
If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything:

    run { extract_evergreens }
Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!

Re: Introducing Drake, a kind of ‘make for data’

#74

Post by aboytsov »

zmmmmm wrote: post #73
aboytsov wrote: post #69

Earlier quoted context omitted.

Thanks for your feedback. We do mention parallelization in the designdoc, it's just not implemented yet. It's quite easy to add though. We have a lot of features spec'ed out, but not implemented. I would appreciate if you elaborated on separating step definitions from dependency definitions. In my mind, they are the same thing. If you mean that steps might not be connected by input-output relationship, but still have…

> I would appreciate if you elaborated on separating step definitions from dependency definitions As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page: evergreens.csv $OUTPUT So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert a…

I see. Thank you very much. I think this is very cool. I can see several problems with this approach, and I would greatly appreciate it if you could comment on that. After all, I don't know Bpipe.

The fundamental issue is why do you have to repeat the filename, and I did give it some thought.

1. What your example does is allows to allocate dependencies based on positions. It's pretty cool. This seems to be easily reproducible in Drake, if we add a special symbol that would just mean "a temporary file" for the output, and "last temporary output" for the input (by the way, you don't need colons):

    _  $OUTPUT

    evergreens.csv  $OUTPUT
or even:

     $OUTPUT

    evergreens.csv  $OUTPUT
2. One of the problems, as you can see, that it only works if you don't care about the filenames, i.e. you use a temporary file. Similarly, your Bpipe expression:

    run { fix_names + extract_evergreens }
doesn't care about filenames as well. How do you add it there? What if you need this file for debugging purposes, or if it's an input to some further step down the road? In this case, you'd have to do what you want to avoid doing (i.e. modify the original step).

3. I'm even more concerned with multiple inputs and multiple outputs. As long as your workflow is simple, you can get away with a + b. But when it's more complicated, you would have to do something like:

    run { (((fix_names + extract_evergreens) * and_some_otheroutput) + some_other_step) * some_other_output }
(I used * as an operator that puts two outputs together to create an input with two files for the next command. Mathematically, + is better for that and * is for what + is used in your examples. :))

As you can see, it gets unreadable so fast, that you'd want to use some sort of identifiers to specify dependencies, and would end up with a scheme pretty much equivalent to filenames. The fact that some file might be a temporary is a related, but parallel problem.

4. Now even worse, I'm not quite sure how this syntax could accomodate multiple outputs. If fix_name creates several outputs, and extract_evergreens uses only one, you can't get around it without some weird syntax and specifying a numeric position. It also gets out of hand pretty quickly and you're back to using some sort of identifiers, be it filenames or not.

5. Speaking of identifiers, you can use variables in Drake instead of filenames, so you can abstract filenames away. But it seems to me there's a more fundamental problem in play.

6. If you're concerned with coupling implementation and input and output names, Drake has methods for this:

    fix_names()
        sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT

    extract_evergreens()
        grep Evergreen $INPUT > $OUTPUT

    renamed.csv
or even, as discussed above:
To summarize, I think your example is cool, but it seems to only be practical for rather simple workflows. And I can also see how Drake can easily be extended to support such syntactic sugar. For more complicated dependencies though, I don't really see a better approach.

I would love to hear your further thoughts on the matter, and whether you'd like to see something similar to what I proposed in Drake. Or something else.

Artem.

Re: Introducing Drake, a kind of ‘make for data’

#75

Post by aboytsov »

tedunangst wrote: post #72
aboytsov wrote: post #66

Earlier quoted context omitted.

Really? This is not a weird request, this is one of the most common things we do when we're developing a workflow. You need to do this every time you make changes to code and you want these changes to propagate. You can't run "make all", because it literally builds everything. You might be working on a specific branch of the workflow, and the overall workflow could be huge. And out-of-date in a lot of places. Or it c…

Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.

Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v...

Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process.

You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results?

When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result.

Makes sense?

Re: Introducing Drake, a kind of ‘make for data’

#76

Post by tedunangst »

aboytsov wrote: post #75
tedunangst wrote: post #72

Earlier quoted context omitted.

Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.

Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v... Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on.…

Thanks.

Re: Introducing Drake, a kind of ‘make for data’

#77

Post by daemon13 »

Artem, the approach you guys are using is really EXCELLENT!

I think that a bit of a disconnect here may be because some OPs might be used to 'compiling' code versus 'compiling' data angle that you are using.

This is especially evident by make dependencies discussion with lars512.

To give a simple specific example: I have a dataset of say 5000-50000 SKUs that are aggregated across 9-12 dimensions. My final report/analysis uses 3 scenarios. Now one sub-set of one scenario has changed [that's the raw input] - of course running 'data compilation' by using data that changed and ONLY what depends on it is the most effective&efficient approach.

Just my 2 financial cents...

Re: Introducing Drake, a kind of ‘make for data’

#78

Post by aboytsov »

daemon13 wrote: post #77

Artem, the approach you guys are using is really EXCELLENT! I think that a bit of a disconnect here may be because some OPs might be used to 'compiling' code versus 'compiling' data angle that you are using. This is especially evident by make dependencies discussion with lars512. To give a simple specific example: I have a dataset of say 5000-50000 SKUs that are aggregated across 9-12 dimensions. My final report/anal…

Thank you very much for your kind words and support, and we certainly are looking forward to your feedback, feature requests and bug reports, as well as your code contributions, should you so desire.

We built this based on our own pain points with a larger audience in mind. We hope we got some things right, because the success of any tool is defined by its users. So, if you like it, let's build a thriving community together!

Artem.

Re: Introducing Drake, a kind of ‘make for data’

#79

Post by wink »

aboytsov wrote: post #71
danpalmer wrote: post #68

Earlier quoted context omitted.

Thanks for the detailed and well explained reply. I have limited experience with Clojure, but it does seem to be a good match to this sort of task due to it's structure. However the JVM seems to be a real drawback to me. Perhaps with something like Scheme or Lisp you might get a similar program structure, and be able to compile to faster binaries? The REPL is a solution, but as many developers are using tools like ma…

I can certainly see your point about using Drake in an automated environment where this delay would still matter, but running a daemon is not practical. I think you have a lot of good arguments against JVM. There were some moments when I thought it might not have been the best choice as well - for example, Java world is notoriously poor with dealing with child processes. So, I agree, but there are several arguments t…

I would say if a startup overhead time of Clojure is sadly a really bad choice for fire-and-forget cli scripts, but "large scale data processing" doesn't fit this criterion for me.

Re: Introducing Drake, a kind of ‘make for data’

#80

Post by pencilcode »

aboytsov wrote: post #45
pencilcode wrote: post #41

Earlier quoted context omitted.

out of curiosity, why did you go the clojure route instead of the scala route? From what i understand, scala has more libraries available, including ai and nlp libraries but maybe my impression is not correct?

It's hard to compare Clojure and Scala. Scala is a multi-paradigm programming language with strong OOP support and functional support. It's arguably more verbose than Clojure but looks much more similar to Java. Clojure is a Lisp. Lisp stands aside all other programming languages, first of all, because it supports syntactic abstraction (a.k.a. "code is data"). Hardcode addicts (I'm not one of them) say there are only…

Thanks for your feedback. I've been playing around with both languages, and was leaning towards scala since it seemed more likely i could use it professionally, even though i liked clojure a bit more, sortta like the lisp like syntax.
Post reply on HN