Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Write Gitlab CI Pipelines in Python Code

gitlab.com

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Write Gitlab CI Pipelines in Python Code

#71

Post by BiteCode_dev »

dvdkon wrote: post #69
BiteCode_dev wrote: post #56

Earlier quoted context omitted.

> Programming in Python is order of magnitude "harder" and error prone than writing a simple YAML file. For a simple config, sure. But as soon as you have something more complicated, you end up with: - a frankenstein monster DSL, with pseudo flow control + funcs backed in through a patched up templating system - weird error messages, no stack trace, and type errors that accumulates - no DRY - no tooling: type checks,…

Slightly relevant: I made a hack to write Ansible playbooks in Python [1]. It doesn't fix everything, for example runtime control flow still has to be described as attributes of tasks, but it does help with parametrising. It's a question whether it's worth it to add such a hack or if I should just make do with YAML. [0]: https://gitlab.com/-/snippets/2112382

It's a great hack actually.

Re: Write Gitlab CI Pipelines in Python Code

#72

Post by erikds »

infogulch wrote: post #45
zmmmmm wrote: post #19

Another step in the endless cycle of configuration vs code. It's not an accident that we are in a deep cycle of constrained configuration languages (yaml/json/etc etc). People chose to go there because before that we had a cycle of using programming languages and people hated it, for all sorts of good reasons. Now I see we are on the way back into adding wrappers around the static config files, to turn them back into…

I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/

I would put my money on lisp.

The structure of data and code is the same.

The result is that you start with configuration. But you all ways have a nice escape hatch

Re: Write Gitlab CI Pipelines in Python Code

#73

Post by orf »

freetime2 wrote: post #52

> We started this project because of Gitlab CI yaml files growing over thousands of lines. Is this typical? All of our Gitlab CI files are well under 100 lines. What sorts of things are these pipelines doing that require so much configuration? Our CI steps are basically: * Build * Run some static analysis * Test * Publish build artifacts With each step taking only a few lines. Most of the “heavy lifting” is managed b…

If you've got thousands of repositories deploying services written in different languages to many different environments and multiple clusters within each environment then your CI files can get quite lengthy. Using Docker helps standardize things, but our shared pipeline repo is still 3.8k lines of YAML.

Re: Write Gitlab CI Pipelines in Python Code

#74

Post by protomikron »

pydry wrote: post #66
infogulch wrote: post #45

Earlier quoted context omitted.

I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/

This is deeply misguided, I think. Dhall is a neat hack, but it's still a hack. Any and all attempts to munge YAML or to turn YAML into something Turing complete are signals that a scripting language would have been a better fit for the problem. The exit to this cycle is: * to use configuration languages for things that are genuinely configuration now and always * scripting language APIs for things that are genuinely…

Dhall is not turing-complete. That's the whole point.

Re: Write Gitlab CI Pipelines in Python Code

#75

Post by rendaw »

zmmmmm wrote: post #19

Another step in the endless cycle of configuration vs code. It's not an accident that we are in a deep cycle of constrained configuration languages (yaml/json/etc etc). People chose to go there because before that we had a cycle of using programming languages and people hated it, for all sorts of good reasons. Now I see we are on the way back into adding wrappers around the static config files, to turn them back into…

> before that we had a cycle of using programming languages

When was that?

Re: Write Gitlab CI Pipelines in Python Code

#76

Post by pydry »

protomikron wrote: post #74
pydry wrote: post #66

Earlier quoted context omitted.

This is deeply misguided, I think. Dhall is a neat hack, but it's still a hack. Any and all attempts to munge YAML or to turn YAML into something Turing complete are signals that a scripting language would have been a better fit for the problem. The exit to this cycle is: * to use configuration languages for things that are genuinely configuration now and always * scripting language APIs for things that are genuinely…

Dhall is not turing-complete. That's the whole point.

That was incidental to dhall. Dhall doesn't and was never intended to help clean up the mess made by having ansible code be Turing complete (for example).

I'm pretty sure the whole point of dhall is to DRY out configuration files and to hack on type guarantees to config files.

I played with it for a bit and those are the only two use cases I could find. Those two use cases usually come encumbered with several others which it does not help with at all, but a scripting language would.

A good, refactored-to-be-DRY config file that is typesafe completely obliviates any need for dhall and if you've got a Turing complete YAML monstrosity it's not going to help much.

It reminds me a bit of XSLT (except that was accidentally Turing complete of course).

Re: Write Gitlab CI Pipelines in Python Code

#77

Post by nly »

vsgzusnex wrote: post #48
nly wrote: post #26

Earlier quoted context omitted.

The problem in C++ land (my land) is all your dependencies are probably using different build systems. Do you wan to migrate all your third party libs to Bazel, or just hack up a few lines of CI yaml to call the authors build system? I lean on gitlab yaml quite heavily and it feels quite effortless

We vendored all of our third party dependencies into our Bazel build. There were a lot and it was a giant PITA but over a multi-year horizon it was worth it. That doesn't mean it's the right choice for every business but it paid off for us.

Vendoring all your third party dependencies is fine, but you still have to build them.

Personally I use the native build system that upstream uses for most of them, and just hack in my compiler flags/options where necessary. Then I use CI triggers to rebuild my projects when the dependencies are rebuilt.

Re: Write Gitlab CI Pipelines in Python Code

#78

Post by sofixa »

xdennis wrote: post #59
sofixa wrote: post #51

Earlier quoted context omitted.

via comments ! YAML is an abomination in itself ( spaces for logic?!), the last thing it needed was comments for extra logic.

Proper spacing makes code readable. Forcing it is a good thing.

Indeed, it does, when done via something like go fmt. But in the case of YAML, a pedantic level of "proper spacing" is used for logic. 2 or 3 spaces shouldn't change the meaning and order of things

Re: Write Gitlab CI Pipelines in Python Code

#79

Post by infogulch »

pydry wrote: post #66
infogulch wrote: post #45

Earlier quoted context omitted.

I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/

This is deeply misguided, I think. Dhall is a neat hack, but it's still a hack. Any and all attempts to munge YAML or to turn YAML into something Turing complete are signals that a scripting language would have been a better fit for the problem. The exit to this cycle is: * to use configuration languages for things that are genuinely configuration now and always * scripting language APIs for things that are genuinely…

You just described the cycle, not a path to exit it. There is no such thing as "genuinely configuration now and always", nor "genuinely not configuration".

Systems change, and requirements of those systems change. Oh now someone wants to do more things with what used to be 'just configuration': scale it up and to work across many instances by scripting configuration templates with parameters. Look, all of this scripted automation is doing the same things that can be neatly divided into buckets, can we simplify it and turn it into a few configuration options? And so the cycle repeats. The churn is not reducible to labeling it as pointless make-work, its born of changing and growing systems.

Re: Write Gitlab CI Pipelines in Python Code

#80

Post by infogulch »

pydry wrote: post #76
protomikron wrote: post #74

Earlier quoted context omitted.

Dhall is not turing-complete. That's the whole point.

That was incidental to dhall. Dhall doesn't and was never intended to help clean up the mess made by having ansible code be Turing complete (for example). I'm pretty sure the whole point of dhall is to DRY out configuration files and to hack on type guarantees to config files. I played with it for a bit and those are the only two use cases I could find. Those two use cases usually come encumbered with several others…

This comment doesn't make any sense to me

> A good, refactored-to-be-DRY config file that is typesafe completely obliviates any need for dhall and if you've got a Turing complete YAML monstrosity it's not going to help much.

Dhall is a "refactored-to-be-DRY config file that is typesafe", and the host of safety-related features ( https://docs.dhall-lang.org/discussions/Safety-guarantees.ht... ) make this practical to implement, even while evaluating untrusted & potentially malicious dhall code. These guarantees are much stronger than what nearly any other config or scripting language provides.

Post reply on HN