Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing Drake, a kind of ‘make for data’

blog.factual.com

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Introducing Drake, a kind of ‘make for data’

#51

Post by aboytsov »

jeffdavis wrote: post #50

Cool project. I expected to be underwhelmed, but when I saw the dependency stuff, I was impressed. Maybe it should include a hook so that it can detect dataset changes automatically by running a separate command (or did I miss it?). With a bit of creativity, I think there may be a lot of applications here.

This is an awesome idea. Currently Drake only supports timestamped and forced evaluations, but it would be great to have an evaluation abstraction where you could provide your own implementation of whether a target's changed and/or whether a target is to be considered fresher/younger than another target. Timestamped would compare modification times, forced would return true, and it could be extended indefinitely.

If you're serious about it, please submit a feature request (https://github.com/Factual/drake/issues), and describe more specifically what you would like to be able to do in your case.

Thank you for a great thought.

Artem.

Re: Introducing Drake, a kind of ‘make for data’

#52

Post by lars512 »

aboytsov wrote: post #47
lars512 wrote: post #46

Earlier quoted context omitted.

Make can certainly generate multiple outputs, and can trivially be coerced to redo any step you like. Provided you add your code as a dependency in the analysis, then it will happily redo only what's changed, giving you nice tight iterations. I think it's real limitations are with multi-machine setups, as in the HDFS problem you're mentioning. Then you need a new tool.

Sorry, I might be very ignorant of make - could you please give me a command to re-build a particular target and everything that depends on it?

So make's default behaviour "make somefile.csv" is to build the whole tree of dependencies. To force rebuild of everything, run "make -B somefile.csv". It then assumes everything is out of date.

To force rebuild of one step, just delete its output or run "touch" on one of its dependencies before running make. Then that step will get redone.

I like to have generated data in a separate folder, say "output/" which you can then snapshot, blow away, or do what you like with. Basically though, I keep it separate from data and code inputs.

Re: Introducing Drake, a kind of ‘make for data’

#53

Post by aboytsov »

lars512 wrote: post #52
aboytsov wrote: post #47

Earlier quoted context omitted.

Sorry, I might be very ignorant of make - could you please give me a command to re-build a particular target and everything that depends on it?

So make's default behaviour "make somefile.csv" is to build the whole tree of dependencies. To force rebuild of everything, run "make -B somefile.csv". It then assumes everything is out of date. To force rebuild of one step, just delete its output or run "touch" on one of its dependencies before running make. Then that step will get redone. I like to have generated data in a separate folder, say "output/" which you c…

Thanks! This much I know. But it doesn't answer my question. Let me repeat it: could you please give me a command to re-build a particular target and everything that depends on it?

Re: Introducing Drake, a kind of ‘make for data’

#54

Post by lars512 »

aboytsov wrote: post #53
lars512 wrote: post #52

Earlier quoted context omitted.

So make's default behaviour "make somefile.csv" is to build the whole tree of dependencies. To force rebuild of everything, run "make -B somefile.csv". It then assumes everything is out of date. To force rebuild of one step, just delete its output or run "touch" on one of its dependencies before running make. Then that step will get redone. I like to have generated data in a separate folder, say "output/" which you c…

Thanks! This much I know. But it doesn't answer my question. Let me repeat it: could you please give me a command to re-build a particular target and everything that depends on it?

That's exactly what "make -B mytarget" does...

Are you thinking of a particular problematic scenario?

Re: Introducing Drake, a kind of ‘make for data’

#55

Post by aboytsov »

lars512 wrote: post #54
aboytsov wrote: post #53

Earlier quoted context omitted.

Thanks! This much I know. But it doesn't answer my question. Let me repeat it: could you please give me a command to re-build a particular target and everything that depends on it?

That's exactly what "make -B mytarget" does... Are you thinking of a particular problematic scenario?

No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget and everything that depends on it. Without rebuilding other parts of the workflow that you don't need.

Re: Introducing Drake, a kind of ‘make for data’

#56

Post by wtracy »

lars512 wrote: post #54
aboytsov wrote: post #53

Earlier quoted context omitted.

Thanks! This much I know. But it doesn't answer my question. Let me repeat it: could you please give me a command to re-build a particular target and everything that depends on it?

That's exactly what "make -B mytarget" does... Are you thinking of a particular problematic scenario?

Aboytsov wants to rebuild the target and everything that depends on the target, not rebuild the target and everything that the target depends on. He wants to walk the dependency tree in the opposite direction.

Re: Introducing Drake, a kind of ‘make for data’

#58

Post by tedunangst »

aboytsov wrote: post #55
lars512 wrote: post #54

Earlier quoted context omitted.

That's exactly what "make -B mytarget" does... Are you thinking of a particular problematic scenario?

No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget and everything that depends on it. Without rebuilding other parts of the workflow that you don't need.

This is a really weird request. make won't rebuild things that haven't changed, so the default make all rule will only rebuild the things depending on mytarget. Every time you change mytarget, just run make (all) and everything that depends on mytarget (and only those things) will be rebuilt.

Re: Introducing Drake, a kind of ‘make for data’

#59

Post by danpalmer »

With an empty workflow, this is the result of `drake --version`.

  $ time drake --version
    Drake Version 0.1.0
    Target not found: ...
    drake --version  5.42s user 0.18s system 188% cpu 2.969 total
For short scripts that you should be running in the shell, this is really bad. I expect basic make commands on small projects to be effectively instant. Compilation might take a bit longer, but 5.4s to print the version points to a 5s overhead on all executions.

I'm guessing this is due to the JVM overhead, so that pretty much says this project isn't suited to the JVM. The JVM is great for long running processes, and applications where the overhead is a very small percentage of the total running time, but if it takes 5s longer than `make` to print it's version, that's really not a good sign.

This is a fantastic idea, and I will definitely be using it. But this overhead needs fixing.

Re: Introducing Drake, a kind of ‘make for data’

#60

Post by lars512 »

aboytsov wrote: post #55
lars512 wrote: post #54

Earlier quoted context omitted.

That's exactly what "make -B mytarget" does... Are you thinking of a particular problematic scenario?

No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget and everything that depends on it. Without rebuilding other parts of the workflow that you don't need.

This is basically the case where you don't (or it's infeasible to) capture the dependences fully, so you want to rebuild everything from target onwards after some change.

In that case: "rm target; make target"

Post reply on HN