Live data from Hacker News

Write Gitlab CI Pipelines in Python Code

gitlab.com

61–70 of 97 posts

Re: Write Gitlab CI Pipelines in Python Code

#61
post #60
post #59

Earlier quoted context omitted.

Proper spacing makes code readable. Forcing it is a good thing.

It's a good thing, until you ask a non-technical user to edit the config file. Then they become confused because a tab is visually, but not functionally equivalent to a number of spaces.

I wouldnt ask a non-technical person to write any of these config files.

If I want non-technical people to configure something I need to write a GUI for self-serving.

Re: Write Gitlab CI Pipelines in Python Code

#62

Gitlab can load its ci file from a remote web server. What I have done is generate the yaml programmatically and this gets returned from the web server. Less than ideal but it at least allows dynamic creation

You can just have a CI job generate a CI config file and have GitLab CI load that into a child pipeline. Look up dynamic pipelines.

I came across this great blog post: https://www.objectif-libre.com/en/blog/2021/02/23/a-new-era-...

Documentation: https://docs.gitlab.com/ee/ci/parent_child_pipelines.html

GitLab 13.10 also added support for parallel matrix job execution in child pipelines, speeding up the execution once more.

https://about.gitlab.com/releases/2021/03/22/gitlab-13-10-re...

There's more to dynamic pipeline creation, collecting blog post ideas in https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11122#... :)

Re: Write Gitlab CI Pipelines in Python Code

#63

It seems like a lot of people don't understand that having a DECLARATIVE language/configuration/whatever such a huge advantage it's insane. It's so easy to write, you can never make logical mistakes, avoid all kind of bugs, it's just easy to learn and you can just write the desired result. I never understood these project where somebody makes a very easy declarative thing and makes it imperative. Programming in Pytho…

> Programming in Python is order of magnitude "harder" and error prone than writing a simple YAML file. For a simple config, sure. But as soon as you have something more complicated, you end up with: - a frankenstein monster DSL, with pseudo flow control + funcs backed in through a patched up templating system - weird error messages, no stack trace, and type errors that accumulates - no DRY - no tooling: type checks,…

You have linting and logger in Ansible. Type checks, kind of depend.

Debugging is definitely something I also would like to see something better coming out of Ansible than what is currently available.

Re: Write Gitlab CI Pipelines in Python Code

#64
post #4

What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Nuke [1] gets close but there are still a lot of tasks that don't have C# bindings, such as publishing build artifacts and uploading test results. While I'm dreaming about my perfect CI, I'd also like the abi…

> What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Every tool I've used allows you to do this. Just write the script in the language of your choice, then run it in the pipeline. A pipeline should be considered like "markup" around your scripts. Everything s…

In practice the YAML seems to be used to hook things up - uploading assets, sending messages on slack, etc.

If gitlab/GitHub could just provide hooks to do this stuff for a range of scripting languages that could be easily dry run this would solve the problem.

Re: Write Gitlab CI Pipelines in Python Code

#65

It's almost like we've turned yaml into some sort of primitive assembly file that must be compiled by a higher-order language. What was once created for human readability, has now turned into a machine generated and machine parsed format.

It was created for human readability, but always with machine interop in mind, otherwise English/, would prob. do just fine in most cases, especially if sprinkled with some markup.

Re: Write Gitlab CI Pipelines in Python Code

#66
post #19

Another step in the endless cycle of configuration vs code. It's not an accident that we are in a deep cycle of constrained configuration languages (yaml/json/etc etc). People chose to go there because before that we had a cycle of using programming languages and people hated it, for all sorts of good reasons. Now I see we are on the way back into adding wrappers around the static config files, to turn them back into…

I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/

This is deeply misguided, I think. Dhall is a neat hack, but it's still a hack.

Any and all attempts to munge YAML or to turn YAML into something Turing complete are signals that a scripting language would have been a better fit for the problem.

The exit to this cycle is:

* to use configuration languages for things that are genuinely configuration now and always

* scripting language APIs for things that are genuinely not configuration (e.g. build systems).

* provide both for domains that are genuinely mixed (a lot of kubernetes probably fits into this category - some things are configuration for some people and dynamic or generated for others).

It sounds simple but it is hard - the correct border between config and code is usually hard to perceive and how your tools end up being used is inherently unpredictable.

Re: Write Gitlab CI Pipelines in Python Code

#67

It seems like a lot of people don't understand that having a DECLARATIVE language/configuration/whatever such a huge advantage it's insane. It's so easy to write, you can never make logical mistakes, avoid all kind of bugs, it's just easy to learn and you can just write the desired result. I never understood these project where somebody makes a very easy declarative thing and makes it imperative. Programming in Pytho…

Have you seen the kind of "config files" YAML is being used nowadays? Gitlab CI, Ansible, Saltstack, various low-code tools... It's no better than a scripting language, because it turns out if what you need is to encode logic and control flow, you can't get away from it by shouting "DECLARATIVE" and pretending the problems don't exist. Pretty much all of these complex tools that use YAML don't use it for declarative purposes, but as a kind of hybrid AST/DSL. You're not avoiding anything except for maybe the need to parse a proper language.

Re: Write Gitlab CI Pipelines in Python Code

#68
post #19

Another step in the endless cycle of configuration vs code. It's not an accident that we are in a deep cycle of constrained configuration languages (yaml/json/etc etc). People chose to go there because before that we had a cycle of using programming languages and people hated it, for all sorts of good reasons. Now I see we are on the way back into adding wrappers around the static config files, to turn them back into…

I get your instinct, but there is one thing missing. I’m theory, my CI isn’t really being run in a known environment. I can’t write a shell script to power the ci, because how am I to know the CI launcher even has a shell? Of course, once an environment has been deployed, I am safe to switch to write shell scripts, calling system commands, etc. But it does make some sense to be completely static at the start and I haven’t be even picked an operating system.

Re: Write Gitlab CI Pipelines in Python Code

#69

It seems like a lot of people don't understand that having a DECLARATIVE language/configuration/whatever such a huge advantage it's insane. It's so easy to write, you can never make logical mistakes, avoid all kind of bugs, it's just easy to learn and you can just write the desired result. I never understood these project where somebody makes a very easy declarative thing and makes it imperative. Programming in Pytho…

> Programming in Python is order of magnitude "harder" and error prone than writing a simple YAML file. For a simple config, sure. But as soon as you have something more complicated, you end up with: - a frankenstein monster DSL, with pseudo flow control + funcs backed in through a patched up templating system - weird error messages, no stack trace, and type errors that accumulates - no DRY - no tooling: type checks,…

Slightly relevant: I made a hack to write Ansible playbooks in Python [1]. It doesn't fix everything, for example runtime control flow still has to be described as attributes of tasks, but it does help with parametrising. It's a question whether it's worth it to add such a hack or if I should just make do with YAML.

[0]: https://gitlab.com/-/snippets/2112382

Post reply on HN