Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dgsh – Directed graph shell

dmst.aueb.gr

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Dgsh – Directed graph shell

#21

Post by nerdponx »

mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

This post is making me think it would be a great educational exercise to construct equivalent data processing flows in some popular tools: Make, Airflow, Luigi, Snakemake, Rake, others?

Re: Dgsh – Directed graph shell

#22

Post by cturner »

mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

Something I have found fun in the past: using xslt where the underlying document is not xml. In order for xslt to work (in java setting, apache libs) you do not need an underlying xml document, just something that satisfies the appropriate java interface. For example, you could wrap a filesystem directory structure.

Re: Dgsh – Directed graph shell

#23

Post by wfunction »

cturner wrote: post #22
mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

Something I have found fun in the past: using xslt where the underlying document is not xml. In order for xslt to work (in java setting, apache libs) you do not need an underlying xml document, just something that satisfies the appropriate java interface. For example, you could wrap a filesystem directory structure.

Is it possible to show what XSLT is and why it's useful in like 5 minutes? I've always wanted a transformation language of some sort, but I've never managed to figure out XSLT (probably because I've never needed it) so I don't know what problems it solves or doesn't solve.

Re: Dgsh – Directed graph shell

#24

Post by visarga »

I write complex shell commands every day, but when it gets longer than 2-3 rows I switch to a text editor and write it in Perl instead. I see no need to use bash up to that complexity, doesn't look good in terminal.

Poorman version of multiple pipes is to write intermediate results into files, then "cat" the files as many times as needed for the following processes. I use short file names "o1", "o2" standing for output-1, output-2 and see them as temp variables.

Re: Dgsh – Directed graph shell

#26

Post by samuell »

mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

Exactly our experience too, from complex machine learning workflows in various aspects of drug discovery.

We basically did not really find any of the popular DSL-based bioinformatics pipeline tools (snakemake, bpipe etc) to fit the bill. Nextflow came close, but in fact allows quite some custom code too.

What worked for us was to use Spotify's Luigi, which is a python library rather than DSL.

The only thing was that we had to develop a flow-based inspired API on top of Luigi's more functional programming based one, in order to make defining dependencies fluent and easy enough to specify for our complex workflows.

Our flow-based inspired Luigi API (SciLuigi) for complex workflows, is available at:

https://github.com/pharmbio/sciluigi

We wrote up a paper on it as well, detailing a lot of the design decisions behind it:

http://dx.doi.org/10.1186/s13321-016-0179-6

Then, lately we are working on a pure Go alternative to Luigi/SciLuigi, since we realized that with the flow-based paradigm, we could just as well just rely on the Go channels and go-routines to create an "implicit scheduler" very simply and robustly. This is work in progress, but a lot of example workflows already work well (it has 3 times less LOC than a recent bioinformatics pipeline tool written in python and put into production). Code available at:

https://github.com/scipipe/scipipe

It is also very much a programming library rather than a DSL.

It in fact even implements streaming via named pipes, seemingly allowing somewhat similar operations as dgsh, with a bit more code probably, but with the (seeming) benefit of a bit easier handling of multiple inputs and outputs (via the flow-based progr. ports concept).

dgsh looks real interesting for simpler operations where there is one main input and output though - which occur a lot for ad-hoc work in the shell, in our experience. Will have to test it out for sure!

Re: Dgsh – Directed graph shell

#27

Post by baq »

samuell wrote: post #26
mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

Exactly our experience too, from complex machine learning workflows in various aspects of drug discovery. We basically did not really find any of the popular DSL-based bioinformatics pipeline tools (snakemake, bpipe etc) to fit the bill. Nextflow came close, but in fact allows quite some custom code too. What worked for us was to use Spotify's Luigi, which is a python library rather than DSL. The only thing was that…

Have you checked out airflow? Any opinions?

Re: Dgsh – Directed graph shell

#28

Post by samuell »

baq wrote: post #27
samuell wrote: post #26

Earlier quoted context omitted.

Exactly our experience too, from complex machine learning workflows in various aspects of drug discovery. We basically did not really find any of the popular DSL-based bioinformatics pipeline tools (snakemake, bpipe etc) to fit the bill. Nextflow came close, but in fact allows quite some custom code too. What worked for us was to use Spotify's Luigi, which is a python library rather than DSL. The only thing was that…

Have you checked out airflow? Any opinions?

I have looked a bit at code examples of Airflow, but was worried that it seems to have a similar problem as a lot of other pipeline tools: That in the main workflow specification, dependencies are specified between tasks only, not between the individual inputs and outputs of each task (between tasks rather than data).

This means that this info needs to be implemented "manually" in some less declarative manner somewhere else, breaking the declarative-ness of the workflow specification.

I have posted about it some time ago here, mentioning AirFlow specifically: http://bionics.it/posts/workflows-dataflow-not-task-deps

Re: Dgsh – Directed graph shell

#29

Post by rcthompson »

elsherbini wrote: post #17
rcthompson wrote: post #16

Earlier quoted context omitted.

Here's one example where I had to use a kind of ugly hack ot make it work with Snakemake, a Python Makefile-style "DAG-of-rules" workflow tool: https://github.com/DarwinAwardWinner/CD4-csaw Basically, I need to first fetch the metadata on all the samples, and then later group them by treatment based on that metadata. In other words, the structure of later parts of the DAG depends on the results of executing earlier p…

I use snakemake quite a bit, it was cool to scan through your Snakefile and learn some things. The processify decorator looks really useful[0]. It's possible that you could use snakemake subworkflows [1] for this issue of "pre-workflow" workflows. [0] https://github.com/DarwinAwardWinner/CD4-csaw/blob/master/pr... [1] https://bitbucket.org/snakemake/snakemake/wiki/Documentation...

I also use a subworkflow in this workflow, but for a different purpose (the subworkflow is also on Github: https://github.com/DarwinAwardWinner/hg38-ref). But subworkflow rules are still resolved as part of the same DAG, so they have the same issue. Hence the need for a separate pre-workflow outside the normal framework of Snakemake.

By the way, I guess I didn't add a comment explaining this, but the reason for using the processify decorator is that the snakemake API is not re-entrant, so calling `snakemake` from within a Snakefile normally breaks things. The solution is to call it in a separate process.

Re: Dgsh – Directed graph shell

#30

Post by timthelion »

mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

I downvoted your comment, because it doesn't seem to me that you read the article and are responding to the contents. You are simply responding with a pre-formed opinion. Conversations only work when you read first, then think, and finally respond. But I guess conversations cannot happen on HN, because everything has to be so FAST in silicone valley.
Post reply on HN