Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dgsh – Directed graph shell

dmst.aueb.gr

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Dgsh – Directed graph shell

#11

Post by mtrn »

steveb wrote: post #10
mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

We've been working on a directed graph execution engine called Converge https://github.com/asteris-llc/converge . In this case the task resource http://converge.aster.is/0.5.0/resources/task/ might help, as it allows you to create a directed graph using any kind of interpreter (for example, Python or Ruby) instead of having to use the DSL.

Nice, thanks for the pointer. It's nice to see templated shell calls, as these can be a powerful bridge between orchestration and execution.

Re: Dgsh – Directed graph shell

#12

Post by mtdewcmu »

This looks like potentially a great tool. It might be helpful if the author showed the code examples alongside the equivalent code in bash, so it's easy to see both what the example code is doing and how much effort is saved by doing it in dgsh.

Re: Dgsh – Directed graph shell

#13

Post by xiaq »

I've always thought about integrating this functionality into elvish https://github.com/elves/elvish but cannot cone up with a good syntax. dgsh has a good one, but unfortunately using & breaks its traditional semantics. Does anyone has some idea of a tradition-compatible grammar?

Also, to nitpick, this is more accurately called a directed acyclic graph shell, or simply a DAG shell. The language doesn't seem to allow cycles. dagsh reads nicer than dgsh too.

Re: Dgsh – Directed graph shell

#14

Post by DenisM »

mtrn wrote: post #7
rtpg wrote: post #6

Earlier quoted context omitted.

I've been thinking about this space a lot too, would you mind listing out some of the messier use cases that you have?

> I've been thinking about this space a lot Me too, for better or for worse. As for the issues, there are many. Just quickly a few: * Data provider has an FTP server, most files are automatically generated, some are hand-named (with inconsistencies). How do you handle (without a lot of effort) a list of exceptions along with the regular files? * Data provider has a good strict XML schema, but the relevant information…

That kind of book would be a great service to humanity. I don't know if you will sell many, but anyone inventing a new ETL tool would be served well by reading it. Perhaps a paper for a journal like ACM would be a better format. Or you could make it into a wiki. Or an "ETL Nightmares monthly" newsletter, with best user submissions.

Re: Dgsh – Directed graph shell

#16

Post by rcthompson »

rtpg wrote: post #6
mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

I've been thinking about this space a lot too, would you mind listing out some of the messier use cases that you have?

Here's one example where I had to use a kind of ugly hack ot make it work with Snakemake, a Python Makefile-style "DAG-of-rules" workflow tool: https://github.com/DarwinAwardWinner/CD4-csaw

Basically, I need to first fetch the metadata on all the samples, and then later group them by treatment based on that metadata. In other words, the structure of later parts of the DAG depends on the results of executing earlier parts of the DAG, so the full structure of the DAG is not known initially. The solution I used was to split the workflow in two: a "pre-workflow workflow" that fetches the sample metadata and then the main workflow which reads the metadata and builds the DAG based on it. See here: https://github.com/DarwinAwardWinner/CD4-csaw/blob/master/Sn...

This a common pattern that I see when putting together bioinformatics workflows: the full DAG of actions to execute cannot be known until part of the way through executing that DAG. Most workflow tools can't handle this gracefully. Another Python DAG-executor, called doit, can handle this case, by specifying that some rules should not be evaluated until after others have finished running. But it doesn't have some features that I wanted from Snakemake (e.g. compute cluster execution), so I ended up with the above solution instead.

Re: Dgsh – Directed graph shell

#17

Post by elsherbini »

rcthompson wrote: post #16
rtpg wrote: post #6

Earlier quoted context omitted.

I've been thinking about this space a lot too, would you mind listing out some of the messier use cases that you have?

Here's one example where I had to use a kind of ugly hack ot make it work with Snakemake, a Python Makefile-style "DAG-of-rules" workflow tool: https://github.com/DarwinAwardWinner/CD4-csaw Basically, I need to first fetch the metadata on all the samples, and then later group them by treatment based on that metadata. In other words, the structure of later parts of the DAG depends on the results of executing earlier p…

I use snakemake quite a bit, it was cool to scan through your Snakefile and learn some things. The processify decorator looks really useful[0].

It's possible that you could use snakemake subworkflows [1] for this issue of "pre-workflow" workflows.

[0] https://github.com/DarwinAwardWinner/CD4-csaw/blob/master/pr...

[1] https://bitbucket.org/snakemake/snakemake/wiki/Documentation...

Re: Dgsh – Directed graph shell

#19

Post by nerdponx »

mtdewcmu wrote: post #12

This looks like potentially a great tool. It might be helpful if the author showed the code examples alongside the equivalent code in bash, so it's easy to see both what the example code is doing and how much effort is saved by doing it in dgsh.

It doesn't look all that different to me. Seems like it's just saving you mess around with assigning function inputs and outputs to shell variables. Otherwise it just looks like piping stuff around between functions.

Re: Dgsh – Directed graph shell

#20

Post by karlmdavis »

This is perhaps a bit off-topic, but what I really wish more data processing/ETL tools supported is the concept of transactional units. Too many of them seem to start with the worldview that "we need to shove in as many of the separate bits as we possibly can."

What's often needed for robust systems, instead, is solid support for error handling such that "if this bit doesn't make it in, then neither does that bit." Data is always messy and dirty, and too many ETL systems don't seem architected to cope with that reality.

Of course, maybe I just haven't found the right tools. Anyone know of tools that handle this particularly well?

Post reply on HN