Dagger: a new way to build CI/CD pipelines

Yes, we use CUE as more than just a YAML replacement. In particular we use CUE references to construct the DAG and "compile" it to buildkit.

And, yes, Dagger will gradually fill the missing values in the CUE tree during runtime. Essentially resolving the DAG on the fly. It is pretty cool :)

We have discussed this topic at length with the CUE developers. Our conclusion is that CUE's deterministic core is what matters, and the `_tool.cue` pattern is more peripheral: more of a reference for how other tools might use cue for non-deterministic operations. It's not realistic for CUE to be both a ubiquitous deterministic language, and a successful non-deterministic tool. Its priority is clearly the former, and we're focusing on the latter.

So much for "devops" merging together development and operations.

"A developer’s dream is a devops engineer’s nightmare" underines the failure of devops.

I remember reading about composable pipelines a while ago. I remember thinking this is going to take the devops world by storm, hopefully Dagger brings us closer to that dream.

Good question. No, we don't intend to replace the CI service. We think of CI as infrastructure, and there are plenty of great infrastructure providers out there. And you are absolutely correct that for Dagger to succeed, it has to remain agnostic to infrastructure providers, which means we cannot build an infrastructure provider business ourselves. If the experience of hosting Dagger becomes so bad that it affects the developer experience, we might ship such a feature as a stopgap and even charge for it - but it is not a strategic priority. The sooner other infrastructure providers offer a fantastic hosting experience, the better. That is not where we see the biggest opportunity for Dagger.

However, there is a great opportunity to help businesses manage their software supply chain. What is running where, and how did it get there? Who is authorized to run which pipelines on which machines, and which which inputs? What is the diff between this dev environment and staging? Staging and production? Etc. Keep in mind this is not limited to your production pipeline; there are staging environments, development environments, all running pipelines all day long. It's hard to simply know everything that is going on.

Each time Dagger runs your pipeline, it can produce telemetry about every aspect of your supply chain. Git SHAs, docker image checksums and cryptographic signatures, specific versions of kubernetes and cloudformation configurations, before and after templating, test results, monitoring status... It also integrates with your secrets management backends, your developer's local source code. Basically every node in your supply chain can be a node in your DAG if you want it to. The logical next step is to give you a centralized place to send all this telemetry, and tools to extract insights from it. You could also perhaps manage the configuration of your various Dagger engines in one location.

Another product that is often requested is a visual DAG debugger. When a pipeline break, you want to know why, and staring at your CI logs is definitely not the best experience for that. With a web UI, there's a lot we can do there.

The business opportunity boils down to this: if CI/CD pipelines are software, there ought to be a platform for developing that software, and an ecosystem of developers creating value around that platform. Dagger aspires to create that missing developer ecosystem. If we succeed, there will be no shortage of business opportunities. If we fail, none of the other features will matter.

Full YAML is crazy, but the StrictYAML subset I can definitely live with.

I regularly go over the main alternatives (TOML, JSON, XML), and they all have serious warts for config IMO.

Better formats exists (such as HOCON) but are much less popular. For a public tool I'd still stick with a YAML subset that everyone knows and can work with a minimum of fuss.