71–80 of 92 posts
Earlier quoted context omitted.
Looping in ansible/terraform is exactly the problem. Logic and control flow in YAML/hcl is a nightmare. Plus there's no debugging! You can't set breakpoints in a yaml file or HCL. Adding more YAML to be parsed by other YAML is just terrible at scale.
You definitely can in Ansible.
Ansible / Terraform are great when you're declarative, but once you start adding too much logic, things get hairy.
Proprietary CI/CD systems are a waste of time. If it's more complicated than a shell script, you need to strip it down.
Earlier quoted context omitted.
>Are Turing-complete languages a bad choice for pipelines? I'd say that non turing complete languages are a bad fit for pipelines. Even mildly complex pipeline will eventually have loops and conditionals. Better python than some originally-a-config-YAML language turned into an imperative monstrosity with loops and conditionals bolted on.
Dagger originally started with CUE, and is still powered by it under the hood, which has the constructs you mention, while also being turing incomplete. I don't understand this move to define infra and CI imperatively, and tool vendors moving to support umptine languages for their users... Say what the world should look like, not how to get there?
It's really a blessing for something like CUE to come (I've had some exposure to gcl/bcl and if it wasn't for the internal diffing tool, I would've been lost - https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1.... )
I've even started prototyping something like bazel/gn but based on cue, but no time to finish it up.
I'm working on a similar project, portable pipelines built with Nim :)
The idea behind this project is to:
1. Dev and Prod pipelines runs the same code, the only difference is where are executed. Allows easy troubleshooting and faster development.
2. Tests output by default on each script.
3. Decouples the code from the image/container. This allows it to be embedded into any CI stack.
4. Complex rules for retries specific scripts rather than the whole pipeline or even the step (stage)
A typical pipeline definition looks like this:
```yaml
kind: pipeline
version: 1
policy: strict
steps:
- hello
hello:
- script: echo "hello world"
expected_output: "hello world"
expected_return_code: 0
retries: 5
```
```bash
./takito --tasks tasks.yml --step hello
```Curious if cuelang just ended up being too much of a hurdle for onboarding. I like it and have used it quite a bit but there's something about the syntax that makes it impenetrable for many.
I had a look at Dagger, but I'm having a lot of success recently just building CI/CD pipelines with Nix. My `.gitlab-ci.yml` file is then just a thin wrapper of `nix build` commands, and pipelines can be run locally in the exact same environment as CI/CD.
Confusing. I initially thought someone ported the Dagger DI framework to Python: https://dagger.dev/
Debugging and maintaining code is to be avoided at all costs. You may prefer writing code as a developer, but it doesn't provide value to the people paying your paycheck, and it doesn't result in better outcomes than simple composeable actions. Proprietary CI/CD systems are a waste of time. If it's more complicated than a shell script, you need to strip it down.
I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…
Personally, it's pretty interesting to see how different tools in the space handle things, even without getting into where and how your pipelines run. It's nice to see the increasing move towards defining everything in "code", regardless of whether it's a DSL or a turing-complete language - I'll take a versioned Jenkinsfile over messing about in the UI most days.
> I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes.
I'd say that YAML is passable on its own, but gets more and more inadequate, the harder the things you're trying to do get.
Thankfully, I've been able to keep most of my CI pipelines relatively simple nowadays, along the lines of:
- initialize some variables that don't come out of the box from the CI solution
- parse or process any files that are needed for the build/action, such as attaching metadata to project description
- do the build (nowadays typically just building an OCI container), or whatever else the pipeline needs to do (since you can do more than just build applications)
- save any build artefacts, push containers, do logging or whatever else is needed
That said, there's basically nothing preventing me or anyone else from including a Python script, or a Go program, or even Bash scripts (if you don't want to think about getting an environment where most of the other languages are available, in lieu of the footguns of Bash) and just running those. Then, control flow, looping, using additional libraries or tools suddenly becomes more easy.
> Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itself right.
In regards to writing correct pipelines, I really liked how GitLab CI lets you validate your configuration and even shows how the pipeline would look like, without executing anything, in their web UI: https://docs.gitlab.com/ee/ci/lint.html I think most tools should have something like that, as well as pipeline visualizations - anything to make using them more user friendly!
As for the cycle times, if most of what the build or CI action (whatever it might be) needs is already described as "code", you should be able to run the steps locally as well, either with a separate wrapper script for the stuff that you won't get locally (like CI injected environment variables, which you can generate yourself), or with a local runner for the CI solution.
This is why I mentioned Drone, which has something nice in this regard: https://docs.drone.io/cli/drone-exec/
But generally, for most simpler pipelines (like the example above), you can even just set up an IDE run profile. In my case, I typically version a few run configurations for JetBrains IDEs, that can build containers for me, run tests and do other things. Sometimes the local experience can be a bit better than what you get on the CI server: if you have any integration tests that automate a browser with Selenium (or a more recent solution), you can essentially sit back and watch the test execute on your machine, instead of having to rely on screenshots/recordings on the server after execution.
Of course, much of this would gradually break down, the more complicated your CI pipelines would become. The only thing I can recommend is KISS: https://en.wikipedia.org/wiki/KISS_principle Sadly, this is a non-solution when you're not in control of much of the process. Here's hoping that Dagger and other solutions can incrementally iterate on these aspects and make CI/CD easier in the future!
Earlier quoted context omitted.
Think about it from the perspective of the provider of the build platform (e.g. CircleCI, GitLab, GitHub). There are way fewer edge cases parsing YAML files than allowing Turing complete languages.
Be that as it may, YAML has plenty of edge cases that make it the wrong choice.