Write Gitlab CI Pipelines in Python Code
gitlab.com
Write Gitlab CI Pipelines in Python Code
1–10 of 97 posts
Re: Write Gitlab CI Pipelines in Python Code
#2Reminds me of the original atlassian bamboo pipeline as code impl that had you write the pipeline as Java or Groovy, can’t remember specifically. They got dunked on for it not being a declarative file.
Now we’ve got a pipeline being defined by a declarative file being generated by code.
Re: Write Gitlab CI Pipelines in Python Code
#3Reminds me of the original atlassian bamboo pipeline as code impl that had you write the pipeline as Java or Groovy, can’t remember specifically. They got dunked on for it not being a declarative file. Now we’ve got a pipeline being defined by a declarative file being generated by code.
We use bamboo at my job, and it's java that generates a yaml file that is then shoved into bamboo. It's a completely shitshow. why would i use a turing complete language to generate a declarative yaml file containing turing complete shell scripts? Why not just write the shell script in the first place?
Re: Write Gitlab CI Pipelines in Python Code
#4What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc.
Nuke [1] gets close but there are still a lot of tasks that don't have C# bindings, such as publishing build artifacts and uploading test results.
While I'm dreaming about my perfect CI, I'd also like the ability to download benchmark results from previous commits so that I can generate trend graphs and publish them in the build results.
To do this right the CI system would have to have an API using REST, GraphQL, gRPC or some such API format that generates clients in many languages. That way they don't have to maintain bindings in every language.
Re: Write Gitlab CI Pipelines in Python Code
#5Reminds me of the original atlassian bamboo pipeline as code impl that had you write the pipeline as Java or Groovy, can’t remember specifically. They got dunked on for it not being a declarative file. Now we’ve got a pipeline being defined by a declarative file being generated by code.
That is interesting. In fact I've googled for it and found a blog post with really nastay Java code defining a pipeline. Of course, in code you can do nasty things and create insane pipelines. Thus I think it is good having both - the declarative base and a generator on top. The declarative base ensures having an easy way to get into pipeline mechanics. However if you plan to write really complex pipelines, the pipeline-as-code approach jumbs in. At this point your'e firm with the basics and know what your'e doing in code.
Amazon is doing the same with the Cloud Development Kit (CDK). With CDK you can code your infrastructure in a number of languages (java, csharp, .net, python, typescript), which was synthesized into cloud formation to be finally deployed. For smaller projects and teams not firm with one of those languages, plain CFN may be much better. However after learning CDK you won't create any infrastructure without it.
Re: Write Gitlab CI Pipelines in Python Code
#6What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Nuke [1] gets close but there are still a lot of tasks that don't have C# bindings, such as publishing build artifacts and uploading test results. While I'm dreaming about my perfect CI, I'd also like the abi…
I think the main issue is that most build systems are not expressive enough to do complex packaging, code gen, etc that most projects need. I've been using Bazel for a few projects/teams and it's worked really well. Once your build system is as expressive as you need you get a lot of freedom. Bazel also let's you execute completely local builds and remote builds triggered locally. If you set this up debugging CI issues is amazingly simple.
Re: Write Gitlab CI Pipelines in Python Code
#7What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Nuke [1] gets close but there are still a lot of tasks that don't have C# bindings, such as publishing build artifacts and uploading test results. While I'm dreaming about my perfect CI, I'd also like the abi…
I think the main issue is that most build systems are not expressive enough to do complex packaging, code gen, etc that most projects need. I've been using Bazel for a few projects/teams and it's worked really well. Once your build system is as expressive as you need you get a lot of freedom. Bazel also let's you execute completely local builds and remote builds triggered locally. If you set this up debugging CI issu…
Totally agree. I've never used Bazel; it sounds a lot like Nuke. I had a coworker whose job was to maintain the CI. His commit messages would look like this sometimes:
> Fix CI issue with blah blah blah
> Hmm that didn't work lets try something from stackoverflow
> Build fix
> Build fix please work
> Please
> I hate my life
> I am tired and hungry, I want to go home
> Stupid yaml
The CTO would call him like a week later to make sure he was okay. This happened often before we switched to Nuke. It rarely happened afterwards. It's awesome being able to debug your CI locally.
Re: Write Gitlab CI Pipelines in Python Code
#8Gitlab can load its ci file from a remote web server. What I have done is generate the yaml programmatically and this gets returned from the web server. Less than ideal but it at least allows dynamic creation
Re: Write Gitlab CI Pipelines in Python Code
#9Earlier quoted context omitted.
I think the main issue is that most build systems are not expressive enough to do complex packaging, code gen, etc that most projects need. I've been using Bazel for a few projects/teams and it's worked really well. Once your build system is as expressive as you need you get a lot of freedom. Bazel also let's you execute completely local builds and remote builds triggered locally. If you set this up debugging CI issu…
Totally agree. I've never used Bazel; it sounds a lot like Nuke. I had a coworker whose job was to maintain the CI. His commit messages would look like this sometimes: > Fix CI issue with blah blah blah > Hmm that didn't work lets try something from stackoverflow > Build fix > Build fix please work > Please > I hate my life > I am tired and hungry, I want to go home > Stupid yaml The CTO would call him like a week la…
Yep, and if you get everything working smoothly enough developers can use the same very polished tooling for their own workflow.
Re: Write Gitlab CI Pipelines in Python Code
#10This is cool, but I'd rather my CI jobs are just k8s or nomad or systemd jobs. Then the code we deploy is the same as the code we use to build, and it doesn't matter WHAT you do in build land, go nuts and do whatever you want.
CI systems will either grow into general purpose code runners or they will wither and die.