Faster and more flexible pipelines with a Directed Acyclic Graph • Hacker News Board

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#31

Post by jayd16 » Tue, May 12, 2020, 7:22 PM UTC

epage wrote: post #30

jayd16 wrote: post #29

Earlier quoted context omitted.

Can't you do something like: -Stage1 needs A, B, C -D needs Stage1

Basically faking a stage by making a job? That helps but the "definition" of a stage is still far away from the job definition when you have enough of them and people won't know to update this when they copy/paste a job definition for adding a new test. Plus, visualization is still terrible :)

I don't disagree. Just trying to help.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#32

Post by [deleted] » Tue, May 12, 2020, 7:26 PM UTC

defen wrote: post #27

tr352 wrote: post #22

Earlier quoted context omitted.

Agreed. First sentence: "A Directed Acyclic Graph will let you run pipeline steps out of order." That's simply false.

Interpret "order" as "the order in which the steps are defined in your YAML file", and it's simply true.

[deleted]

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#33

Post by tr352 » Tue, May 12, 2020, 7:42 PM UTC

defen wrote: post #27

tr352 wrote: post #22

Earlier quoted context omitted.

Agreed. First sentence: "A Directed Acyclic Graph will let you run pipeline steps out of order." That's simply false.

Interpret "order" as "the order in which the steps are defined in your YAML file", and it's simply true.

I'm nitpicking but my point is that a DAG doesn't let you run anything. It might represent something, though, such as a bunch of tasks and a "must be executed before" relation. It's sloppy, imprecise use of language.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#34

Post by deckar01 » Tue, May 12, 2020, 8:58 PM UTC

lindsayolson wrote: post #19

deckar01 wrote: post #15

This feature is currently broken. It can let deploys run without builds/tests passing. https://gitlab.com/gitlab-org/gitlab/-/issues/31526

Hi! GitLab Community Advocate here: I noticed the issue you linked is closed. I believe the conversation is continued here: https://gitlab.com/gitlab-org/gitlab/-/issues/213080 Hope that is helpful. If so, feel free to add a comment so the team can see.

Due to that issue being closed maintainers assumed it was fixed and went around closing related issues without testing if the problems were fixed.

https://gitlab.com/gitlab-org/gitlab/issues/198570#note_2936...

I have had a really negative experience reporting bugs to GitLab in the last 12 months. I have spent tons of time investigating bugs in new features only to have the devs ignore it. It feels like they are sweeping things under the rug so they can hit deadlines for officially announcing new features.

https://gitlab.com/gitlab-org/gitlab/-/issues/39534#note_283...

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#35

Post by donmcronald » Tue, May 12, 2020, 9:11 PM UTC

xyzzy_plugh wrote: post #3

I didn't read too deeply, but I'm glad to see all these sorts of systems slowly converge on a sane model. The part that saddens me is that they all end up looking like a glorified distributed make.

I'd hardly call any of them sane. There used to be debates about checking compilers into the VCS and now no one bats an eye when the entire build system is constantly churning underneath you. Today you have GitLab Runner 12.x. Next month it'll be 13.x. I imagine most other CI systems are similar, the hosted variants being especially bad because you can't even decide when to update.

Modern CI feels like a shell script using a bunch of external APIs got tossed into a blender and poured into a YAML config. Then everyone claims builds are repeatable because they run in a container, but ignore the fact that if your Dockerfile has a single command that isn't working with exact versions for resources (apt-get anyone?) you can't rely on the build being repeatable between jobs in the same pipeline.

A big problem I think we'll face in the future is there's no value in giving developers a product they can control. GitLab CI will evolve so you need a top tier plan to do anything interesting and they'll eventually own your build environment. GitHub will push things like Codespaces and they'll own your development environment. The idea that anyone would even touch GitHub Actions blows my mind.

What happened to the developers that want to control their environments? Do you actually own your codebase if it's useless without a bunch of paid subscriptions you're using as part of your development workflow? What happens when your CI provider triples the price or shuts down entirely?

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#36

Post by ahuang1018 » Wed, May 13, 2020, 7:53 AM UTC

deckar01 wrote: post #34

lindsayolson wrote: post #19

Earlier quoted context omitted.

Hi! GitLab Community Advocate here: I noticed the issue you linked is closed. I believe the conversation is continued here: https://gitlab.com/gitlab-org/gitlab/-/issues/213080 Hope that is helpful. If so, feel free to add a comment so the team can see.

Due to that issue being closed maintainers assumed it was fixed and went around closing related issues without testing if the problems were fixed. https://gitlab.com/gitlab-org/gitlab/issues/198570#note_2936... I have had a really negative experience reporting bugs to GitLab in the last 12 months. I have spent tons of time investigating bugs in new features only to have the devs ignore it. It feels like they are swee…

Hi! GitLab Community Advocate here - I agree that we need to get better how we improve the quality and performance of existing features/bugs. I am sorry for the negative experience you had. Please ping me on any long standing issues/bugs that are important to you and I'll do my best to route it to the directly responsible individuals.

For the new features part, going fast is a core part of our strategy, as is focusing on breadth over depth (https://about.gitlab.com/company/strategy/#breadth-over-dept...). Not only does this provide an easier path to collaborate and contribute for the wider community, it allows us to shorten the feedback loop with everyone that uses GitLab so that we can invest our time and effort in the areas that matter most to the wider community.Hope this is helpful.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#37

Post by BorisTheBrave » Wed, May 13, 2020, 8:55 AM UTC

tr352 wrote: post #33

defen wrote: post #27

Earlier quoted context omitted.

Interpret "order" as "the order in which the steps are defined in your YAML file", and it's simply true.

I'm nitpicking but my point is that a DAG doesn't let you run anything. It might represent something, though, such as a bunch of tasks and a "must be executed before" relation. It's sloppy, imprecise use of language.

By that logic, no code lets you run anything, it just "represents" a program that actually runs. That's just pedantry.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#38

Post by tr352 » Wed, May 13, 2020, 9:51 AM UTC

BorisTheBrave wrote: post #37

tr352 wrote: post #33

Earlier quoted context omitted.

I'm nitpicking but my point is that a DAG doesn't let you run anything. It might represent something, though, such as a bunch of tasks and a "must be executed before" relation. It's sloppy, imprecise use of language.

By that logic, no code lets you run anything, it just "represents" a program that actually runs. That's just pedantry.

Code runs and possibly lets you run something (for example in case of an OS or interpreter). I don't see a problem there.

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#39

Post by furkanayhan » Wed, May 13, 2020, 10:11 AM UTC

deckar01 wrote: post #34

lindsayolson wrote: post #19

Earlier quoted context omitted.

Hi! GitLab Community Advocate here: I noticed the issue you linked is closed. I believe the conversation is continued here: https://gitlab.com/gitlab-org/gitlab/-/issues/213080 Hope that is helpful. If so, feel free to add a comment so the team can see.

Due to that issue being closed maintainers assumed it was fixed and went around closing related issues without testing if the problems were fixed. https://gitlab.com/gitlab-org/gitlab/issues/198570#note_2936... I have had a really negative experience reporting bugs to GitLab in the last 12 months. I have spent tons of time investigating bugs in new features only to have the devs ignore it. It feels like they are swee…

Hello Jared!

GitLab Senior Backend Engineer - Verify (CI) here - Thanks for reaching out! I'll try to explain current situation of these issues.

1. https://gitlab.com/gitlab-org/gitlab/-/issues/198570

We know that this bug with DAG/needs is quite annoying. That's why we've been actively discussing about this bug in other issue: https://gitlab.com/gitlab-org/gitlab/-/issues/213080

Can you please take a look at the discussions there?

2. https://gitlab.com/gitlab-org/gitlab/-/issues/39534

Unfortunately we had no choice but to give an option to fix this behavior, instead of fixing it. That's because, if we fixed this bug immediately, it would probably break existing pipelines.

That's why we allowed users to fix this behavior themselves (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24605).

Hope this is helpful and please feel free to contact us if you have any questions/feedbacks. Thanks!

Re: Faster and more flexible pipelines with a Directed Acyclic Graph

#40

Post by deckar01 » Wed, May 13, 2020, 4:03 PM UTC

furkanayhan wrote: post #39

deckar01 wrote: post #34

Earlier quoted context omitted.

Due to that issue being closed maintainers assumed it was fixed and went around closing related issues without testing if the problems were fixed. https://gitlab.com/gitlab-org/gitlab/issues/198570#note_2936... I have had a really negative experience reporting bugs to GitLab in the last 12 months. I have spent tons of time investigating bugs in new features only to have the devs ignore it. It feels like they are swee…

Hello Jared! GitLab Senior Backend Engineer - Verify (CI) here - Thanks for reaching out! I'll try to explain current situation of these issues. 1. https://gitlab.com/gitlab-org/gitlab/-/issues/198570 We know that this bug with DAG/needs is quite annoying. That's why we've been actively discussing about this bug in other issue: https://gitlab.com/gitlab-org/gitlab/-/issues/213080 Can you please take a look at the dis…

These two issues illustrate that GitLab has a quality control problem. If devs aren't dog fooding these features, the community should be given more time to beta test them before the APIs are declared stable.