Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Write Gitlab CI Pipelines in Python Code

gitlab.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Write Gitlab CI Pipelines in Python Code

#81

Post by kortex »

pas wrote: post #55
kortex wrote: post #32

Earlier quoted context omitted.

The big difference is that config files are, at some point after interpolation and templating, static data structures composed of primitive data types. Further, these configs usually act as declarative languages, giving directives to pipelines, controllers, etc., typically with idempotent side-effects. This is just good software engineering. You have some static data structure which describes some result you want to…

Exactly! And this is perfectly achievable with high-level expressive languages too. The important part is that the interface has to have the same semantics as a config file: atomic/transactional. For example the builder pattern does this. You can enjoy all the expressiveness and power of the high-level language, but at some point you create the final config/pipeline/job-graph, call whatever needs to be called, and th…

Yup. My main gripe with Ansible is exactly this. Too much mutable state, too imperative, too much reliance on "gathering facts". There's no singular source of truth. It tries to be declarative, but every Ansible deployment I've maintained (granted, not many) always accretes corner cases and config drift. Maybe that's just the nature of the domain - mutating live systems will always be more imperative than (re)deploying the whole thing from scratch.

I'd much rather use python/jinja/gomplate/what-have-you, do all my fancy expressive stuff with code I can check in, and get a single snapshot of plain, structured text (which I can also check in). Not to mention locking down your invariants (I can't recall how many times I've gotten bit by interpolating docker image uris with the wrong TAG). It's actually pushed me towards first Make and then later python cli tools (cause Makefile syntax is...ugh) to generate configs, rather than relying on any runtime business logic, cause it's just so much saner, traceable, auditable, etc.

Re: Write Gitlab CI Pipelines in Python Code

#82

Post by infogulch »

erikds wrote: post #72
infogulch wrote: post #45

Earlier quoted context omitted.

I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/

I would put my money on lisp. The structure of data and code is the same. The result is that you start with configuration. But you all ways have a nice escape hatch

Too much escape hatch. I would prefer my configuration to be less than turing complete, tyvm.

Re: Write Gitlab CI Pipelines in Python Code

#84

Post by pydry »

infogulch wrote: post #80
pydry wrote: post #76

Earlier quoted context omitted.

That was incidental to dhall. Dhall doesn't and was never intended to help clean up the mess made by having ansible code be Turing complete (for example). I'm pretty sure the whole point of dhall is to DRY out configuration files and to hack on type guarantees to config files. I played with it for a bit and those are the only two use cases I could find. Those two use cases usually come encumbered with several others…

This comment doesn't make any sense to me > A good, refactored-to-be-DRY config file that is typesafe completely obliviates any need for dhall and if you've got a Turing complete YAML monstrosity it's not going to help much. Dhall is a "refactored-to-be-DRY config file that is typesafe", and the host of safety-related features ( https://docs.dhall-lang.org/discussions/Safety-guarantees.ht... ) make this practical to…

The point is you could all get that with YAML without plonking dhall on top if you had a properly designed schema.

Hence it's a hack to circumvent badly designed / no schema.

Re: Write Gitlab CI Pipelines in Python Code

#85

Post by VOBOSHI »

StavrosK wrote: post #13
nerdbaggy wrote: post #8

Gitlab can load its ci file from a remote web server. What I have done is generate the yaml programmatically and this gets returned from the web server. Less than ideal but it at least allows dynamic creation

You can just have a CI job generate a CI config file and have GitLab CI load that into a child pipeline. Look up dynamic pipelines.

Our shared pipeline used to create child pipelines and we wouldn't recommend them.

If all the shared pipeline jobs show up in your consumer project pipeline, you can easily extend them or overwrite them entirely.

Re: Write Gitlab CI Pipelines in Python Code

#86

Post by VOBOSHI »

nerdbaggy wrote: post #17
StavrosK wrote: post #13

Earlier quoted context omitted.

You can just have a CI job generate a CI config file and have GitLab CI load that into a child pipeline. Look up dynamic pipelines.

I feel like there was some weird edge cases I was having with child pipelines. Maybe it was something with artifacts, don’t remember.

Artifacts, passing environment variables from the main pipeline to the child pipeline, expanding variable references in the child pipeline and getting unexpected results...

We're migrating our shared pipeline away from child pipelines. Maybe one day we'll do a write-up. :)

Re: Write Gitlab CI Pipelines in Python Code

#87

Post by zie »

cortesoft wrote: post #16
zie wrote: post #10

This is cool, but I'd rather my CI jobs are just k8s or nomad or systemd jobs. Then the code we deploy is the same as the code we use to build, and it doesn't matter WHAT you do in build land, go nuts and do whatever you want. CI systems will either grow into general purpose code runners or they will wither and die.

Gitlab CI is already basically that

Can I run random long jobs, like say a PostgreSQL instance under it? and keep it monitored and sane?

Re: Write Gitlab CI Pipelines in Python Code

#88

Post by infogulch »

pydry wrote: post #84
infogulch wrote: post #80

Earlier quoted context omitted.

This comment doesn't make any sense to me > A good, refactored-to-be-DRY config file that is typesafe completely obliviates any need for dhall and if you've got a Turing complete YAML monstrosity it's not going to help much. Dhall is a "refactored-to-be-DRY config file that is typesafe", and the host of safety-related features ( https://docs.dhall-lang.org/discussions/Safety-guarantees.ht... ) make this practical to…

The point is you could all get that with YAML without plonking dhall on top if you had a properly designed schema. Hence it's a hack to circumvent badly designed / no schema.

Your argument is akin to "why do you need a high-level language? just write it in assembly, C is just a hack to circumvent badly designed assembly". If a 'properly designed yaml schema' was sufficiently powerful for all these use-cases then why does everyone and their third cousin come up with new configuration languages every week? Clearly they do, thus I must reject the null hypothesis and conclude that plain yaml schemas are not sufficient.

These arguments aren't holding any water.

Re: Write Gitlab CI Pipelines in Python Code

#89

Post by infogulch »

pydry wrote: post #84
infogulch wrote: post #80

Earlier quoted context omitted.

This comment doesn't make any sense to me > A good, refactored-to-be-DRY config file that is typesafe completely obliviates any need for dhall and if you've got a Turing complete YAML monstrosity it's not going to help much. Dhall is a "refactored-to-be-DRY config file that is typesafe", and the host of safety-related features ( https://docs.dhall-lang.org/discussions/Safety-guarantees.ht... ) make this practical to…

The point is you could all get that with YAML without plonking dhall on top if you had a properly designed schema. Hence it's a hack to circumvent badly designed / no schema.

Lets look at a specific example. Take Kubernetes: everything is yaml, with complete schemas, all the way down. From your perspective this is configuration utopia, right? Meanwhile back in reality k8s is the poster child of "yaml hell". From the day it was released, people took one look at it, gave it a giant NOPE and instantly spawned half a dozen templating languages. The most popular of these is helm, which has a terrible, no good, very bad design: full of potential injection attacks from purely textual string substitution, manually specified indentation to embed parameterized blocks, virtually no intermediate validation, no way to validate unused features, etc etc

Compare to dhall which publishes a complete set of dhall-k8s schema mappings which enables you to factor out any design you want down to as few configuration variables as you like, while validating the configuration generators themselves at design time. https://github.com/dhall-lang/dhall-kubernetes#more-modular-...

Re: Write Gitlab CI Pipelines in Python Code

#90

Post by pydry »

infogulch wrote: post #89
pydry wrote: post #84

Earlier quoted context omitted.

The point is you could all get that with YAML without plonking dhall on top if you had a properly designed schema. Hence it's a hack to circumvent badly designed / no schema.

Lets look at a specific example. Take Kubernetes: everything is yaml, with complete schemas, all the way down. From your perspective this is configuration utopia, right? Meanwhile back in reality k8s is the poster child of "yaml hell". From the day it was released, people took one look at it, gave it a giant NOPE and instantly spawned half a dozen templating languages. The most popular of these is helm, which has a t…

>Lets look at a specific example. Take Kubernetes: everything is yaml, with complete schemas, all the way down. From your perspective this is configuration utopia, right?

LOL! No, their schemas are horribly designed and half of their YAML should really be APIs.

But it's hella popular.

>Compare to dhall

Which isn't.

Post reply on HN