Live data from Hacker News

Faster and more flexible pipelines with a Directed Acyclic Graph

about.gitlab.com

11–20 of 40 posts

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#11
post #6

I'm confused, a DAG is a model for the relationships between data. What makes this a separate feature for the product? And it's pretty fundamental to the problem you try to solve, usually, so what were they using before, a list? Tree?

They were using a list-of-sets; the CI pipeline is composed of stages which run in series, each of which runs a set of jobs in parallel. The new dependency DAG allows jobs to explicitly declare dependencies on each other and avoid waiting at stage boundaries.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#12
post #6

I'm confused, a DAG is a model for the relationships between data. What makes this a separate feature for the product? And it's pretty fundamental to the problem you try to solve, usually, so what were they using before, a list? Tree?

> I'm confused, a DAG is a model for the relationships between data. Here they're talking about a DAG as a model for the relationships between computation rather than data . Many systems that model computation use DAGs. For example it's how many compilers understand your code when you compile it.

> Here they're talking about ... computation rather than data.

The lisp people would like a word...

In seriousness, computation is data, and this is perhaps especially obvious in a CI context where your “computation” is increasingly bash scripts in some string field inside of a YAML file.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#13

Earlier quoted context omitted.

> I'm confused, a DAG is a model for the relationships between data. Here they're talking about a DAG as a model for the relationships between computation rather than data . Many systems that model computation use DAGs. For example it's how many compilers understand your code when you compile it.

> Here they're talking about ... computation rather than data. The lisp people would like a word... In seriousness, computation is data, and this is perhaps especially obvious in a CI context where your “computation” is increasingly bash scripts in some string field inside of a YAML file.

> The lisp people would like a word...

Why do you feel the need to say things in such a snarky way as this, as if I'm ignorant?

That was the point - that computation can also be represented in a 'data' structure like this. I already gave that exact example - a compiler representing computation as data.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#14
post #7
post #6

I'm confused, a DAG is a model for the relationships between data. What makes this a separate feature for the product? And it's pretty fundamental to the problem you try to solve, usually, so what were they using before, a list? Tree?

Which part of the explanation of what they mean in the article is unclear?

The article is clear on what they're doing. My confusion is to whether this is a feature announcement or a technical blog on the implementation details.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#16

Earlier quoted context omitted.

> Here they're talking about ... computation rather than data. The lisp people would like a word... In seriousness, computation is data, and this is perhaps especially obvious in a CI context where your “computation” is increasingly bash scripts in some string field inside of a YAML file.

> The lisp people would like a word... Why do you feel the need to say things in such a snarky way as this, as if I'm ignorant? That was the point - that computation can also be represented in a 'data' structure like this. I already gave that exact example - a compiler representing computation as data.

I think they're replying that way because you seem to miss my point, I didn't think I needed to say that "the operations are the data" explicitly.

My confusion was not on what they were doing or why they chose to use a DAG, but why they're presenting it this way and why they weren't using a DAG to begin with.

Like I've written task runners in various forms before. They've always been based on DAGs and topological sorting from the beginning. I wouldn't advertise it as such because the choice of model is obvious to me, I'd be more interested in reading about alternatives that provide other benefits.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#17

Earlier quoted context omitted.

> Here they're talking about ... computation rather than data. The lisp people would like a word... In seriousness, computation is data, and this is perhaps especially obvious in a CI context where your “computation” is increasingly bash scripts in some string field inside of a YAML file.

> The lisp people would like a word... Why do you feel the need to say things in such a snarky way as this, as if I'm ignorant? That was the point - that computation can also be represented in a 'data' structure like this. I already gave that exact example - a compiler representing computation as data.

Whoa. I wasn't trying to be snarky; if anything it was intended to be a humorous jab at how "computation is data" is such a thing in the lisp community.

I agree that your example illustrates the "computation is data" concept; I was just reconciling that with the apparent contradiction ("computation rather than data") earlier in your post.

I wasn't trying to call "gotcha" on you or anything; sometimes I say things that are unclear and I appreciate it when others help me clarify.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#18

So, just like Concourse, Drone and any other modern CI system. About time.

Yeah so I just want to point out that most other CI systems are harder to understand, less documented, harder to bring your own hardware to bear on, and do less for the general software community than GitLab does.

I've used CircleCI, Travis, Drone (I don't think I've used Concourse actually) -- and I by far prefer GitLab CI, it is more featureful and more easy to use.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#19

This feature is currently broken. It can let deploys run without builds/tests passing. https://gitlab.com/gitlab-org/gitlab/-/issues/31526

Hi! GitLab Community Advocate here: I noticed the issue you linked is closed. I believe the conversation is continued here: https://gitlab.com/gitlab-org/gitlab/-/issues/213080

Hope that is helpful. If so, feel free to add a comment so the team can see.

Post reply on HN