Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Drake, a kind of ‘make for data’

blog.factual.com

Post reply on HN
Print view View original

101–110 of 111 posts

Re: Introducing Drake, a kind of ‘make for data’

#101

Post by danpalmer »

wink wrote: post #79
aboytsov wrote: post #71

Earlier quoted context omitted.

I can certainly see your point about using Drake in an automated environment where this delay would still matter, but running a daemon is not practical. I think you have a lot of good arguments against JVM. There were some moments when I thought it might not have been the best choice as well - for example, Java world is notoriously poor with dealing with child processes. So, I agree, but there are several arguments t…

I would say if a startup overhead time of Clojure is sadly a really bad choice for fire-and-forget cli scripts, but "large scale data processing" doesn't fit this criterion for me.

I'm mostly going to use this for parsing XML into some other formats and getting it into SQLite databases I think. The reason I would like to use Drake over 'raw' Python scripts is because it supports a lot of the mundane stuff that goes around the actual processing of the data, and I want to automate the processes.

I typically deal with sub-100MB XML documents, so processing them takes very little time, but having the quick iteration of changing the format and re-outputting is a key part of the development cycle for me, and I think very useful when you are experimenting with new data and seeing how it could be used. Doing quick transforms is awesome.

Re: Introducing Drake, a kind of ‘make for data’

#102

Post by danpalmer »

aboytsov wrote: post #85
abraininavat wrote: post #83

Earlier quoted context omitted.

Did you consider ClojureScript and V8? Are there downsides to ClojureScript that would lead you to use Clojure instead?

To be honest with you, no, we didn't seriously consider it. Maybe we should have. I do not know if ClojureScript would be able to work with all the dependencies we have (for example, Hadoop client library to talk to HDFS). But it's a good point nevertheless. I'll mention it in https://github.com/Factual/drake/issues/1 . Thanks!

I didn't realise originally that Drake integrated with HDFS. Thats a really awesome feature, and I can see why the JVM made sense in development because of existing HDFS libraries.

Re: Introducing Drake, a kind of ‘make for data’

#103

Post by fnbr »

aboytsov wrote: post #100
fnbr wrote: post #98

Earlier quoted context omitted.

Ah, sorry, I should have been more clear. I've actually gone through the readme a few times, to no avail. I'll triple-check it though.

Read the "Installation" section, there's "A nicer way to run Drake" subsection. But I would advise to read the whole "Installation" section carefully.

I actually did that, several times.

My mistake was that I didn't realize I was supposed to have Drake.jar in the same folder as the workflow that I was trying to execute (I'd keep getting the error 'Unable to access jarfile drake.jar'). Naive error, I suppose.

However, I'm still having trouble executing the 'A nicer way to run Drake' instructions. I created a file named 'drake' on my path, and inserted the given text. However, I keep getting the error

'Exception in thread "main" java.lang.NoClassDefFoundError: drake/core'

Was I supposed to alter the script in any way? I just naively copy/pasted.

Re: Introducing Drake, a kind of ‘make for data’

#104

Post by aboytsov »

fnbr wrote: post #103
aboytsov wrote: post #100

Earlier quoted context omitted.

Read the "Installation" section, there's "A nicer way to run Drake" subsection. But I would advise to read the whole "Installation" section carefully.

I actually did that, several times. My mistake was that I didn't realize I was supposed to have Drake.jar in the same folder as the workflow that I was trying to execute (I'd keep getting the error 'Unable to access jarfile drake.jar'). Naive error, I suppose. However, I'm still having trouble executing the 'A nicer way to run Drake' instructions. I created a file named 'drake' on my path, and inserted the given text…

You don't have to have Drake.jar in the same folder as the workflow you're trying to execute.

You create the script as described in the documentation, and you put it somewhere on your PATH along with the JAR file. The JAR files has to be in the same directory as the script.

Sorry if it wasn't clear. I'll fix the doc.

Re: Introducing Drake, a kind of ‘make for data’

#105

Post by aboytsov »

danpalmer wrote: post #101
wink wrote: post #79

Earlier quoted context omitted.

I would say if a startup overhead time of Clojure is sadly a really bad choice for fire-and-forget cli scripts, but "large scale data processing" doesn't fit this criterion for me.

I'm mostly going to use this for parsing XML into some other formats and getting it into SQLite databases I think. The reason I would like to use Drake over 'raw' Python scripts is because it supports a lot of the mundane stuff that goes around the actual processing of the data, and I want to automate the processes. I typically deal with sub-100MB XML documents, so processing them takes very little time, but having t…

Drip now works with Drake! Yes, it's still less than ideal if you're calling Drake hundreds of times from an automated script which you need to run quickly, but for interactive development, it should work just fine:

https://github.com/Factual/drake/wiki/Faster-startup:-Drake-...

Re: Introducing Drake, a kind of ‘make for data’

#106

Post by aboytsov »

aboytsov wrote: post #104
fnbr wrote: post #103

Earlier quoted context omitted.

I actually did that, several times. My mistake was that I didn't realize I was supposed to have Drake.jar in the same folder as the workflow that I was trying to execute (I'd keep getting the error 'Unable to access jarfile drake.jar'). Naive error, I suppose. However, I'm still having trouble executing the 'A nicer way to run Drake' instructions. I created a file named 'drake' on my path, and inserted the given text…

You don't have to have Drake.jar in the same folder as the workflow you're trying to execute. You create the script as described in the documentation, and you put it somewhere on your PATH along with the JAR file. The JAR files has to be in the same directory as the script. Sorry if it wasn't clear. I'll fix the doc.

Actually, it was in the doc. If you followed the instructions below precisely, just send us your terminal log so that we can see what you're missing.

A nicer way to run Drake

We recommend you "install" Drake in your environment so that you can run it by just typing "drake". Here's a convenience script you can put on your path:

  #!/bin/bash
  java -cp $(dirname $0)/drake.jar drake.core $@
Save that as `drake`, then do `chmod 755 drake`. Move the uberjar to be in the same directory. Now you can just type `drake` to run Drake from anywhere.

Re: Introducing Drake, a kind of ‘make for data’

#107

Post by fnbr »

aboytsov wrote: post #106
aboytsov wrote: post #104

Earlier quoted context omitted.

You don't have to have Drake.jar in the same folder as the workflow you're trying to execute. You create the script as described in the documentation, and you put it somewhere on your PATH along with the JAR file. The JAR files has to be in the same directory as the script. Sorry if it wasn't clear. I'll fix the doc.

Actually, it was in the doc. If you followed the instructions below precisely, just send us your terminal log so that we can see what you're missing. A nicer way to run Drake We recommend you "install" Drake in your environment so that you can run it by just typing "drake". Here's a convenience script you can put on your path: #!/bin/bash java -cp $(dirname $0)/drake.jar drake.core $@ Save that as `drake`, then do `c…

I'm embarrassed- you're completely right. My apologies. It's working perfectly now. Thanks for putting up with me!

Re: Introducing Drake, a kind of ‘make for data’

#108

Post by abraininavat »

aboytsov wrote: post #85
abraininavat wrote: post #83

Earlier quoted context omitted.

Did you consider ClojureScript and V8? Are there downsides to ClojureScript that would lead you to use Clojure instead?

To be honest with you, no, we didn't seriously consider it. Maybe we should have. I do not know if ClojureScript would be able to work with all the dependencies we have (for example, Hadoop client library to talk to HDFS). But it's a good point nevertheless. I'll mention it in https://github.com/Factual/drake/issues/1 . Thanks!

Thanks for the response! I ask because I have an idea for a CLI program, and I want to write it in Clojure, but I'm worried about the startup time of the JVM. As I understand it, this issue is mitigated in Drake by the fact that a typical job will crunch lots of data and therefore take lots of time. That's not the case for my program, it needs to be quick.

Re: Introducing Drake, a kind of ‘make for data’

#109

Post by aboytsov »

abraininavat wrote: post #108
aboytsov wrote: post #85

Earlier quoted context omitted.

To be honest with you, no, we didn't seriously consider it. Maybe we should have. I do not know if ClojureScript would be able to work with all the dependencies we have (for example, Hadoop client library to talk to HDFS). But it's a good point nevertheless. I'll mention it in https://github.com/Factual/drake/issues/1 . Thanks!

Thanks for the response! I ask because I have an idea for a CLI program, and I want to write it in Clojure, but I'm worried about the startup time of the JVM. As I understand it, this issue is mitigated in Drake by the fact that a typical job will crunch lots of data and therefore take lots of time. That's not the case for my program, it needs to be quick.

Yes, startup times are a pain. As of this morning, Drake now works with Drip, which is a nifty tool to bring down start up times. It spins "backup" JVMs, so next time you run the command, JVM is ready. It works great for interactive environments where at least several seconds pass between runs, but won't do much if you need to run Drake several times per second from an automated script.

Another option is Nailgun, but it has its limitations, too.

None if this is ideal. If you want to write a very simple CLI program, keep this in mind. You may want to stay away from JVM.

Re: Introducing Drake, a kind of ‘make for data’

#110

Post by zmmmmm »

aboytsov wrote: post #99
zmmmmm wrote: post #97

Earlier quoted context omitted.

I'm sorry I don't have time to answer in full. I'm just going to respond to this one point because I think it's pretty fundamental and perhaps explaining it will clear up other things! > The dependency graph specifies what > steps depend on what steps. If you don't > know it, you don't even know how to > start evaluating the workflow, because > you don't know which step to build > first. I don't understand this state…

> I'm sorry I don't have time to answer in full. We're not getting anywhere. Just give me goddamn examples! :) Please! Examples! > I can see this is really really hard to grok if you're basing everything on the idea of a DAG, and so many tools are that it's very natural to think you couldn't do it any other way. There is no other way . BPipe is based on the idea of a DAG. You just don't see it. > In Bpipe the user de…

> There is no other way. BPipe is based on the idea of a DAG. You just don't see it.

So if you think Bpipe uses a DAG, then I wonder how you would think it deals with:

  run { fix_names + fix_names + fix_names }
In terms of the pipeline stages that run this is cyclic, so it cannot be a DAG. On the other hand the files created do usually form a DAG dependency relationship, but even there, in the most general case, it's not at all impossible in an imperative pipeline to read a file in and write the same file out again in modified form (or more likely, to modify it in place), so the file depends on itself - another non-DAG relationship. I'm sure you'll object to this in a purist sense, and tell me it is a horribly broken idea, but as a practising bioinformatician, when I have a 10TB file and modifying it in place will save me hours and huge amounts of space, I'm much more interested in getting my job done than being pure about things.

I think you're right that we're at diminishing returns here, and I'm sorry I've frustrated you. We're trying to bite off more than we can chew in a forum like this.

I wish you all the best with Drake and I'll definitely check it out down the track (when it supports parallelism, since that's too important to me right now). For now, though, I don't intend to read / respond to any more replies in this thread.

Post reply on HN