Seems like more stuff that Nix solves out of the box
[deleted]
Dagger: a new way to build CI/CD pipelines
71–80 of 267 posts
Re: Dagger: a new way to build CI/CD pipelines
#72We should definitely escape from YAML, but this language isn't it
Re: Dagger: a new way to build CI/CD pipelines
#73Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since, you know... Docker. If you have any questions, I'll be happy to answer them here!
Hi! I've browsed the docs quickly, and I have a few questions. Seems to assume that all CI/CD workflows work in a single container at a time pattern. How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? Not familiar with cue, but can I import/define a common action that is used across multiple jobs? For example on GitHub I…
> Seems to assume that all CI/CD workflows work in a single container at a time pattern.
Dagger runs your workflows as a DAG, where each node is an action running in its own container. The dependency graph is detected automatically, and all containers that can be parallelized (based on their dependencies) will be parallelized. If you specify 10 actions to run, and they don't depend on each other, they will all run in parallel.
> How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation?
It is possible, but not yet convenient (you need to connect to an external docker engine, via a docker CLI wrapped in a container) We are working on a more pleasant API that will support long-running containers (like your test DB) and more advanced synchronization primitives (wait for an action; terminate; etc.)
This is discussed in the following issues:
- https://github.com/dagger/dagger/issues/1337
- https://github.com/dagger/dagger/issues/1249
- https://github.com/dagger/dagger/issues/1248
> Not familiar with cue, but can I import/define a common action that is used across multiple jobs?
Yes! That is one of the most important features. CUE has a complete packaging system, and we support it natively.
For example here is our "standard library" of CUE packages: https://github.com/dagger/dagger/tree/main/pkg
> For example on GitHub I get to duplicate the dependency installation/caching/build across various jobs. (yes, I'm aware that now you can makeshift on GitHub a composite action to reuse)
Yes code reuse across projects is where Dagger really shines, thanks to CUE + the portable nature of the buildkit API.
Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :)
> Can you do conditional execution of actions based on passed in input value/env variable?
Yes, that is supported.
> Any public roadmap of upcoming features?
For now we rely on raw Github issues, with some labels for crude prioritization. But we started using the new Github projects beta (which is a layer over issues), and plan to open that to the community as well.
Generally, we develop Dagger in the open. Even as a team, we use public Discord channels (text and voice) by default, unless there is a specific reason not to (confidential information, etc.)
Re: Dagger: a new way to build CI/CD pipelines
#74I’m ready to be an early (&& paying) adopter of this today, but I have real work to get done so it would be extremely helpful if you could describe what doesn’t work well about your solution today.
Re: Dagger: a new way to build CI/CD pipelines
#75Hell yes. A build system that I can test locally without hacks (like act) is literally the dream.
Re: Dagger: a new way to build CI/CD pipelines
#76Earlier quoted context omitted.
Thank you. It isn't hard to find, but my point was that if you say "using an intuitive declarative language" then a developer will get excited by a code snippet that shows that intuitive developer language. It wasn't there, and I think their post could be improved by having less fluffy language and more code, if they are targeting me, that is. This link does show the code, but it took a few clicks to get there from y…
The language is CUE, which I think will see mass adoption in config / DevOps in the coming years. So regardless what you think of the language today, it is likely to become important and part of your like in the not too distant future. https://cuelang.org | https://cuetorials.com Dagger builds on top of CUE and the (DAG) flow engine therein
That's great context. That should be added to that doc link right at the top, it would have me feel much more safe about investing the time to learn it!
Re: Dagger: a new way to build CI/CD pipelines
#77Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since, you know... Docker. If you have any questions, I'll be happy to answer them here!
What's the monetization strategy going to be?
There will be an optional cloud service (not yet available). Its features will be based on requests from the community. Some problems just can't be solved with a command-line client. For example: visualization of your pipelines; a centralized audit log of all jobs run across all machine; centrally managed access control and policies; etc.
We will not rely on unusual licences to restrict competitors from running Dagger as a service. We don't need to, since `dagger` is a client tool. We do encourage cloud providers to run buildkit as a service, though :)
Generally, we take inspiration from Red Hat for their balancing of open-source community and business. They are very open with their code, and tightly control how their trademark is used. You can clone their IP and use it to compete with them - but you have to build your own brand, and you can't confuse and fragment the Dagger developer community itself. We think that is a fair model, and we are using it as a general guideline.
Re: Dagger: a new way to build CI/CD pipelines
#78How would this handle CI/CD jobs which need to be run on a mac?
From what I understand, everything needs to be a container, and that doesn't work for xcode.
Re: Dagger: a new way to build CI/CD pipelines
#79I played with a similar idea a while ago: https://github.com/ecordell/cuezel/ (cuezel as in: "Bazel but with CUE"), but I was never sure that what I was doing was in the spirit of CUE.
CUE pushes nondeterminism into "_tool.cue"[0] files that are allowed to do things like IO and run external processes. Tool files scratch a similar itch to Makefiles, but they lack an integrated plugin system like Bazel (hence why I played with the idea of CUE + Bazel).
With Dagger you seem to be restricted to the set of things that the dagger tool can interpret, just like with my Cuezel tool you are limited to what I happened to implement.
In CUE `_tool` files you are also limited to the set of things that the tool builtins provide, but the difference is that you know that the rest of the CUE program is deterministic/pure (everything not in a _tool file).
There's clearly value in tooling that reads CUE definitions, and dagger is the first commercial interest in CUE that I've seen, which is exciting.
But I'm most interested in some CUE-interpreter meta-tool that would allow you to import cue definitions + their interpreters and version them together, but for use in `_tool` files to keep the delineation clear. Maybe this is where dagger is heading? (if so it wasn't clear from the docs)
Re: Dagger: a new way to build CI/CD pipelines
#80After reading this entire post, I’m still left wondering what problem this solves for me, beyond fluffy promises of ‘everything is going to be better’. At the very least I’d want to see a comparison with what we have now, to show me how this is better. I get that I can try to explore more, but if I don’t get a compelling reason to do so after reading the introductory post, I’m not very motivated to do so Any motivati…
It mostly solves this problem:
- write code
- run tests
- commit code
- update CI
- commit
- CI broken
- update CI
- commit
- CI broken
- update CI
- ...
The workarounds for this are generally awful.
For Jenkins, you stage your own instance locally and configure your webhooks to use that. It's exactly as terrible as it sounds, and I never recommend this approach.
For Travis and Concourse (I think), you can use their CLI to spin up a runner locally and run your CI/CD yaml against it. It works "fine," as long as you're okay with the runner it creates being different from the runners it actually uses in their environment (and especially your self-hosted runners).
In GitHub Actions, you can use Act to create a Dockerized runner with your own image which parses your YAML file and does what you want. This actually works quite well and is something that threatens Dagger IMO.
Other CI systems that I've used don't have an answer for this very grating problem.
Another lower-order problem Dagger appears to solve is using a markup language to express higher-level constructs like loops, conditionals, and relationships. They're using CUE to do this, though I'm not sure if hiring the creator of BCL (Borgmon Configuration Language) was the move. BCL was notoriously difficult to pickup, despite being very powerful and flexible. I say "lower-order" because many CI systems have decent-enough constructs for these, and this isn't something I'd consider a killer feature.
I also _also_ like that it assumes Dockerized runners by default, as every other CI product still relies on VMs for build work. VMs are useful for bigger projects that have a lot of tight-knit dependencies, but for most projects out there, Dockerized runners are fine, and are often a pain to get going with in CI (though this has changed over the years).