Live data from Hacker News

Dgsh – Directed graph shell

www2.dmst.aueb.gr

11–20 of 61 posts

Re: Dgsh – Directed graph shell

#11
post #3

This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.

I respect Python but the upgrade to Python 3 showed that data processing workloads that can be handled by standard Unix tooling should stay there.

The upgrade was a nightmare for so many organizations. It shouldn't be that way but it was.

Re: Dgsh – Directed graph shell

#12
post #10

This would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh

Murex has had this capability for years. (https://github.com/lmorg/murex)

I’m on my phone at the moment and cooking so cannot type any examples, but if I get time, I’ll throw together some comparisons later tonight

Re: Dgsh – Directed graph shell

#13
post #12
post #10

This would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh

Murex has had this capability for years. ( https://github.com/lmorg/murex ) I’m on my phone at the moment and cooking so cannot type any examples, but if I get time, I’ll throw together some comparisons later tonight

I could not find any mention of DAGs or directed acyclic graphs in the documentation.

Re: Dgsh – Directed graph shell

#14
post #3

This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.

spawning shell commands and the equivalent of piping is surprisingly hard in python. It's almost easier to do in C

There are probably libraries that could help, but then you need to install dependencies which is sad in python for other reasons

Re: Dgsh – Directed graph shell

#15
post #3

This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.

From a glance, it looks like very similar tradeoffs vs bash. Much harder to read in a medium-large application, but much more ergonomic IO and process control.

I.e. much faster to use dgsh for a basic processing DAG, much more painful to use dgsh for a large ETL pipeline.

Python with something like Prefect isn't something you'd use a REPL to bang out a one-off on, but it'd be more maintainable. dgsh would let you use a REPL to bang out a quick and dirty DAG.

Re: Dgsh – Directed graph shell

#16
post #3

This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.

There is a lot of stuff for Python which follows the "express computation as a dag" approach, especially Apache Airflow https://airflow.apache.org/

Apache Airflow solves a very different problem. Its DAGs are static dependencies between sequentially executed processing steps, whereas the DAGs of dgsh express live direct data flows.

Re: Dgsh – Directed graph shell

#17
post #6

Earlier quoted context omitted.

English is my third language and I can confirm I didn't even think about this

Same english is my fourth language and it didn't even appear to me

Author of dgsh here. This is definitely not what I had in mind.

Re: Dgsh – Directed graph shell

#18
post #10

This would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh

Well, the project started 12 years ago (as sgsh), so that fits into your 10-20 years ago window :)

Re: Dgsh – Directed graph shell

#19
post #13
post #12

Earlier quoted context omitted.

Murex has had this capability for years. ( https://github.com/lmorg/murex ) I’m on my phone at the moment and cooking so cannot type any examples, but if I get time, I’ll throw together some comparisons later tonight

I could not find any mention of DAGs or directed acyclic graphs in the documentation.

Yeah it’s not technically DAG since it uses iteration, but then dgsh will use iteration under the hood too.

However Murex does support CSP-style concurrency. So while there’s no syntax sugar for writing graphs, you can very easily create adhoc pipes and pass them around instead of using stdout / stderr.

So it wouldn’t actually take much to refine that with some DAG-friendly syntax.

In fact maybe that can be my next project…

Re: Dgsh – Directed graph shell

#20
post #10

This would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh

I went through two iterations before adopting the current syntax. Truth is neither me nor Doug McIlroy, the inventor of Unix pipes, who kindly and generously provided feedback during dgsh's development, had something better to propose.

What syntax would you propose?

Post reply on HN