Interesting. What are the benefits of thinking of data pipelines in terms of a DAG? Why cant it be cyclical with exit conditions?
Dgsh – Directed graph shell
41–50 of 61 posts
Re: Dgsh – Directed graph shell
#42Earlier quoted context omitted.
There is a lot of stuff for Python which follows the "express computation as a dag" approach, especially Apache Airflow https://airflow.apache.org/
Apache Airflow solves a very different problem. Its DAGs are static dependencies between sequentially executed processing steps, whereas the DAGs of dgsh express live direct data flows.
Do you mean to say that two non-dependant tasks in an Airflow DAG aren't able to concurrently execute? Thats not my experience. I'm also confused by the use of 'static' in this context.
Re: Dgsh – Directed graph shell
#43Hello. In English this makes me think of the phrase “dog shit”. Not sure if that’s intentional or not.
That's what I think when I hear "bash".
That's literally the word for shit in Norwegian: bæsj
https://upload.wikimedia.org/wikipedia/commons/6/69/Nb-b%C3%...
Re: Dgsh – Directed graph shell
#44Earlier quoted context omitted.
Thank you for the suggestion. This would mean that you'd also then create some mapping from each name (like git_log) to its implementation, right?
Yes, using shell functions: git_log() { git log --pretty=tformat:'%at %ae' } Separating function definitions allows you to run, test, and re-use them.
And, more importantly, assign a name to a process, so that it can appear multiple times in the graph.
Re: Dgsh – Directed graph shell
#45Interesting. What are the benefits of thinking of data pipelines in terms of a DAG? Why cant it be cyclical with exit conditions?
A nicer syntax and a lower probability of deadlocks.
Re: Dgsh – Directed graph shell
#46Earlier quoted context omitted.
Apache Airflow solves a very different problem. Its DAGs are static dependencies between sequentially executed processing steps, whereas the DAGs of dgsh express live direct data flows.
Do you mean to say that two non-dependant tasks in an Airflow DAG aren't able to concurrently execute? Thats not my experience. I'm also confused by the use of 'static' in this context.
That's the point: non-dependant tasks can run concurrently in Airflow. In sh/BAsh/dgsh dependant tasks can also run concurrently, as in tar cf - . | xz.
Re: Dgsh – Directed graph shell
#47As someone who loves graphs and Neo4j I wish I had though of this.
Re: Dgsh – Directed graph shell
#48Earlier quoted context omitted.
Yes, using shell functions: git_log() { git log --pretty=tformat:'%at %ae' } Separating function definitions allows you to run, test, and re-use them.
And, more importantly, assign a name to a process, so that it can appear multiple times in the graph.
You might want to try looking at the Neo4j query language Cypher for some possible inspiration for the syntax.
Re: Dgsh – Directed graph shell
#49Hello. In English this makes me think of the phrase “dog shit”. Not sure if that’s intentional or not.
> In English this makes me think of the phrase “dog shit”.
In English, this makes me think of the phrase "dig shell". I guess we just have different things on our minds...
:-p
Re: Dgsh – Directed graph shell
#50Earlier quoted context omitted.
Do you mean to say that two non-dependant tasks in an Airflow DAG aren't able to concurrently execute? Thats not my experience. I'm also confused by the use of 'static' in this context.
That's the point: non-dependant tasks can run concurrently in Airflow. In sh/BAsh/dgsh dependant tasks can also run concurrently, as in tar cf - . | xz.
Ok. thank you!