Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dgsh – Directed graph shell

dmst.aueb.gr

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Dgsh – Directed graph shell

#31

Post by samuell »

nerdponx wrote: post #21
mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

This post is making me think it would be a great educational exercise to construct equivalent data processing flows in some popular tools: Make, Airflow, Luigi, Snakemake, Rake, others?

Indeed, not only for education, but also as a tool to evaluate tools for various use cases, I think. Have been thinking the same and looked hard for anything like a set of evaluation workflows, incorporating various specific "motifs" if you like (such as nested parameter sweeps).

Unfortunately haven't found anything, so for our use cases in bioinformatics, I basically took an example workflow that was used in a course in next-gen sequencing analysis as a starting point:

https://github.com/NBISweden/workflow-tools-evaluation/tree/...

Only partly implemented it in Common Workflow Language [1] and SciPipe [2] so far ... the implementation turned out to take a tremendous of work :P

Much interested if anyone has found / created a more general such set of example workflows.

[1] http://commonwl.org

[2] https://github.com/scipipe/scipipe

Re: Dgsh – Directed graph shell

#32

Post by samuell »

timthelion wrote: post #30
mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

I downvoted your comment, because it doesn't seem to me that you read the article and are responding to the contents. You are simply responding with a pre-formed opinion. Conversations only work when you read first, then think, and finally respond. But I guess conversations cannot happen on HN, because everything has to be so FAST in silicone valley.

This is not really a post. Rather a documentation website. Not sure if it makes sense to have to read through the full documentation to make any comment.

Re: Dgsh – Directed graph shell

#34

Post by baq »

samuell wrote: post #28
baq wrote: post #27

Earlier quoted context omitted.

Have you checked out airflow? Any opinions?

I have looked a bit at code examples of Airflow, but was worried that it seems to have a similar problem as a lot of other pipeline tools: That in the main workflow specification, dependencies are specified between tasks only, not between the individual inputs and outputs of each task (between tasks rather than data). This means that this info needs to be implemented "manually" in some less declarative manner somewhe…

thanks!

Re: Dgsh – Directed graph shell

#35

Post by fatihpense »

wfunction wrote: post #23
cturner wrote: post #22

Earlier quoted context omitted.

Something I have found fun in the past: using xslt where the underlying document is not xml. In order for xslt to work (in java setting, apache libs) you do not need an underlying xml document, just something that satisfies the appropriate java interface. For example, you could wrap a filesystem directory structure.

Is it possible to show what XSLT is and why it's useful in like 5 minutes? I've always wanted a transformation language of some sort, but I've never managed to figure out XSLT (probably because I've never needed it) so I don't know what problems it solves or doesn't solve.

For example, It is very easy to wrap some XML in other XML. Selecting XML nodes with XPath is also powerful. You don't have to write boilerplate Java etc. However, its template logic has a learning curve and it is only useful in work related to XML.

Re: Dgsh – Directed graph shell

#37

Post by vinceguidry »

visarga wrote: post #24

I write complex shell commands every day, but when it gets longer than 2-3 rows I switch to a text editor and write it in Perl instead. I see no need to use bash up to that complexity, doesn't look good in terminal. Poorman version of multiple pipes is to write intermediate results into files, then "cat" the files as many times as needed for the following processes. I use short file names "o1", "o2" standing for outp…

This is what it comes down to to me too. Using the shell to do programming seems to me like putting your job on hard mode.

When I had to do a lot of data processing at my last job, I started building up tools in Ruby. If I had time, I'd hack the workflow so that the next time I needed it, I could just run the tool from the command line.

Eventually I had a pluggable architecture that I could use to pull data from any number of sources and mix it with any other data. Do that with a shell? Why?

Re: Dgsh – Directed graph shell

#38

Post by dwhitena »

mtrn wrote: post #4

I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice…

Thanks for sharing your experience. I work with Pachyderm, which is an open source data pipelining and data versioning framework. Some things like might be relevant to this conversation are the fact that Pachyderm is language agnostic and that it keeps analyses in sync with data (because it triggers off of commits to data versioning). This makes it distinct from Airflow or Luigi, for example.

Re: Dgsh – Directed graph shell

#39

Post by nerdponx »

samuell wrote: post #31
nerdponx wrote: post #21

Earlier quoted context omitted.

This post is making me think it would be a great educational exercise to construct equivalent data processing flows in some popular tools: Make, Airflow, Luigi, Snakemake, Rake, others?

Indeed, not only for education, but also as a tool to evaluate tools for various use cases, I think. Have been thinking the same and looked hard for anything like a set of evaluation workflows, incorporating various specific "motifs" if you like (such as nested parameter sweeps). Unfortunately haven't found anything, so for our use cases in bioinformatics, I basically took an example workflow that was used in a cours…

Yes, thank you! I'll see if maybe I can throw something similar together for a social science data project, like a Titanic dataset run-through.

Re: Dgsh – Directed graph shell

#40

Post by mtdewcmu »

nerdponx wrote: post #19
mtdewcmu wrote: post #12

This looks like potentially a great tool. It might be helpful if the author showed the code examples alongside the equivalent code in bash, so it's easy to see both what the example code is doing and how much effort is saved by doing it in dgsh.

It doesn't look all that different to me. Seems like it's just saving you mess around with assigning function inputs and outputs to shell variables. Otherwise it just looks like piping stuff around between functions.

I think my main reason for posting was to suggest showing the equivalent bash so it was easier to see what the tool did. I just threw in the "looks potentially great" as a little sweetener. ;)
Post reply on HN