This is funny to me since essentially all build/test/etc systems in the past 40+ years have been based around a DAG model.
They are rediscovering the basic features of Make et al.
Faster and more flexible pipelines with a Directed Acyclic Graph
21–30 of 40 posts
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#22I'm confused, a DAG is a model for the relationships between data. What makes this a separate feature for the product? And it's pretty fundamental to the problem you try to solve, usually, so what were they using before, a list? Tree?
Agreed. First sentence: "A Directed Acyclic Graph will let you run pipeline steps out of order." That's simply false.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#23This is funny to me since essentially all build/test/etc systems in the past 40+ years have been based around a DAG model. They are rediscovering the basic features of Make et al.
The fact that everyone goes out of their way to reinvent this really goes to show how awful the Make experience is.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#24I didn't read too deeply, but I'm glad to see all these sorts of systems slowly converge on a sane model. The part that saddens me is that they all end up looking like a glorified distributed make.
Make is a 44 year old implementation of the DAG model. It is not the only possible one, and it has plenty of flaws. At least in principle, there is room for a better implementation of the DAG model, and even different implementations tailored for different ecosystems.
The Mother of all Demos was amazing 5 decades ago. And yet we're not stuck using the exact same UI toolkit as Douglas Engelbart.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#25My problems with this feature:
- You can't have a DAG of stages. Sometimes, especially with a monorepo which this touts, the maintenance of the "needs" becomes burdensome and you want to just block on a stage rather than explicitly named jobs
- The visualization of this is subpar.
So far my favorite implementation of this feature is Azure Pipelines. No idea if its coped into Github Actions yet or not (haven't switched over). I hate that is the case because of my underlying caution about Microsoft after the 90s and early 2000s though supposedly they are better now.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#26This is funny to me since essentially all build/test/etc systems in the past 40+ years have been based around a DAG model. They are rediscovering the basic features of Make et al.
The fact that everyone goes out of their way to reinvent this really goes to show how awful the Make experience is.
Unless you're working on something complex, Make is pretty great. My only complaint as a regular use is the lack of support for spaces in target names.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#27I'm confused, a DAG is a model for the relationships between data. What makes this a separate feature for the product? And it's pretty fundamental to the problem you try to solve, usually, so what were they using before, a list? Tree?
Agreed. First sentence: "A Directed Acyclic Graph will let you run pipeline steps out of order." That's simply false.
Interpret "order" as "the order in which the steps are defined in your YAML file", and it's simply true.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#28This is funny to me since essentially all build/test/etc systems in the past 40+ years have been based around a DAG model. They are rediscovering the basic features of Make et al.
The fact that everyone goes out of their way to reinvent this really goes to show how awful the Make experience is.
I have a pretty awesome experience with make. If you use it for what it's designed for i.e. building files from a graph of dependencies, it's pretty handy. I use for auto-generated code, binary code as well as tasks that depend on other tasks, such as logging into system A requires getting this key from AWS etc. Most people have bash shortcuts for simple scripts to automate things, I instead write Makefiles, since it seems closer to my mental model, you want to do X but doing X depends on resources Y Z T, make figures out the rest.
Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#29My problems with this feature: - You can't have a DAG of stages. Sometimes, especially with a monorepo which this touts, the maintenance of the "needs" becomes burdensome and you want to just block on a stage rather than explicitly named jobs - The visualization of this is subpar. So far my favorite implementation of this feature is Azure Pipelines. No idea if its coped into Github Actions yet or not (haven't switche…
Can't you do something like:
-Stage1 needs A, B, C
-D needs Stage1Re: Faster and more flexible pipelines with a Directed Acyclic Graph
#30My problems with this feature: - You can't have a DAG of stages. Sometimes, especially with a monorepo which this touts, the maintenance of the "needs" becomes burdensome and you want to just block on a stage rather than explicitly named jobs - The visualization of this is subpar. So far my favorite implementation of this feature is Azure Pipelines. No idea if its coped into Github Actions yet or not (haven't switche…
Can't you do something like: -Stage1 needs A, B, C -D needs Stage1
Basically faking a stage by making a job?
That helps but the "definition" of a stage is still far away from the job definition when you have enough of them and people won't know to update this when they copy/paste a job definition for adding a new test.
Plus, visualization is still terrible :)