Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dagger: a new way to build CI/CD pipelines

dagger.io

Post reply on HN
Print view View original

81–90 of 267 posts

Re: Dagger: a new way to build CI/CD pipelines

#81

Post by BillSaysThis »

shykes wrote: post #37
Aeolun wrote: post #24

This bait and switch in the docs when trying to see an example for anything other than Github feels a bit off to me: > If you would like us to document GitLab next, vote for it here: dagger#1677 If you don’t have an example for a specific tool, just don’t add it to your documentation.

To clarify, you can run Dagger on Gitlab today. It just requires some manual configuration that we would like to automate away, for convenience. We have only done this for Github so far, and would like to do it for more. We will look for a way to make this more clear in the documentation.

What about Bitbucket?

Re: Dagger: a new way to build CI/CD pipelines

#83

Post by shykes »

evancordell wrote: post #79

I played with a similar idea a while ago: https://github.com/ecordell/cuezel/ (cuezel as in: "Bazel but with CUE"), but I was never sure that what I was doing was in the spirit of CUE. CUE pushes nondeterminism into "_tool.cue"[0] files that are allowed to do things like IO and run external processes. Tool files scratch a similar itch to Makefiles, but they lack an integrated plugin system like Bazel (hence why I pla…

I know exactly what you mean. Earlier versions of Dagger actually took this "embedding" approach: any CUE configuration is a valid Dagger configuration, with an extra layer of annotation that describes how to make that configuration runnable. What we learned is that this model is thoroughly confusing to almost everyone except hardcore CUE enthusiasts.

Now we use CUE in a more straightforward way: as a superior replacement for YAML. There's a (simple) schema that all Dagger plans must follow; beyond that, you can import any CUE package and apply any definition into your plan. But you don't need to go upstream and annotate these packages with additional Dagger metadata.

I'm not sure if my explanation is clear - even I get confused by this embedding business sometimes.

Re: Dagger: a new way to build CI/CD pipelines

#85

Post by jacques_chester »

nunez wrote: post #80
Aeolun wrote: post #16

After reading this entire post, I’m still left wondering what problem this solves for me, beyond fluffy promises of ‘everything is going to be better’. At the very least I’d want to see a comparison with what we have now, to show me how this is better. I get that I can try to explore more, but if I don’t get a compelling reason to do so after reading the introductory post, I’m not very motivated to do so Any motivati…

It mostly solves this problem: - write code - run tests - commit code - update CI - commit - CI broken - update CI - commit - CI broken - update CI - ... The workarounds for this are generally awful. For Jenkins, you stage your own instance locally and configure your webhooks to use that. It's exactly as terrible as it sounds, and I never recommend this approach. For Travis and Concourse (I think), you can use their…

Concourse has `fly execute` which makes the commit-push-curse problem go away. It's had it since 2015 or so.

Re: Dagger: a new way to build CI/CD pipelines

#86

Post by shykes »

xmlninja wrote: post #84

"A developer’s dream is a devops engineer’s nightmare" Well, the opposite is very much true as well. A devops engineer's dream is usually a developers nightmare.

That is sadly often true today, but not inevitable! When you are overwhelmed with complexity, a natural defense mechanism is to restrict choice. But with better tooling, you can better manage and reduce complexity, which allows you to say "yes" to developers without compromising on reliability and security. That is what we are trying to enable with Dagger.

Re: Dagger: a new way to build CI/CD pipelines

#87

Post by evancordell »

shykes wrote: post #83
evancordell wrote: post #79

I played with a similar idea a while ago: https://github.com/ecordell/cuezel/ (cuezel as in: "Bazel but with CUE"), but I was never sure that what I was doing was in the spirit of CUE. CUE pushes nondeterminism into "_tool.cue"[0] files that are allowed to do things like IO and run external processes. Tool files scratch a similar itch to Makefiles, but they lack an integrated plugin system like Bazel (hence why I pla…

I know exactly what you mean. Earlier versions of Dagger actually took this "embedding" approach: any CUE configuration is a valid Dagger configuration, with an extra layer of annotation that describes how to make that configuration runnable. What we learned is that this model is thoroughly confusing to almost everyone except hardcore CUE enthusiasts. Now we use CUE in a more straightforward way: as a superior replac…

In the examples it looked as though there is still a CUE unification loop happening during dagger processing:

  deploy: netlify.#Deploy & {
    contents: build.contents.output
  }
It looks like dagger is using cue as a bit more than a YAML replacement; it hydrates cue values as it runs - which is cool! - but that's the part that seemed at odds with CUE's philosophy of pushing nondeterminism into clearly marked files.

Re: Dagger: a new way to build CI/CD pipelines

#88

Post by shykes »

BillSaysThis wrote: post #81
shykes wrote: post #37

Earlier quoted context omitted.

To clarify, you can run Dagger on Gitlab today. It just requires some manual configuration that we would like to automate away, for convenience. We have only done this for Github so far, and would like to do it for more. We will look for a way to make this more clear in the documentation.

What about Bitbucket?

The only dependency for running Dagger is buildkit - which itself requires a docker-compatible container runtime. This could be containerd, runc or of course Docker engine. If your environment can run Docker containers, it can run Dagger.

There's another aspect which affects performance: cache persistence. Buildkit needs a place to persist its cache across runs. By default, it relies on local storage. If that is wiped between runs (common in CI environments), everything will still run, it will just be slow - possibly very slow. Luckily, buildkit supports a variety of cache backends, including a Docker registry. In practice, persisting the cache is the most labor-intensive part of integrating in your CI, even though it's technically not a hard dependency. This is the part that we want to automate away for users, but it requires additional engineering work - hence our asking the community for input on what to prioritize.

Re: Dagger: a new way to build CI/CD pipelines

#89

Post by mhitza »

shykes wrote: post #73
mhitza wrote: post #67

Earlier quoted context omitted.

Hi! I've browsed the docs quickly, and I have a few questions. Seems to assume that all CI/CD workflows work in a single container at a time pattern. How about testing when I need to spin up an associated database container for my e2e tests. Is it possible, and just omitted from the documentation? Not familiar with cue, but can I import/define a common action that is used across multiple jobs? For example on GitHub I…

> Seems to assume that all CI/CD workflows work in a single container at a time pattern. Dagger runs your workflows as a DAG, where each node is an action running in its own container. The dependency graph is detected automatically, and all containers that can be parallelized (based on their dependencies) will be parallelized. If you specify 10 actions to run, and they don't depend on each other, they will all run in…

Thank you for the detailed response. I appreciate you taking the time. One last question/note.

> Note: you won't need to configure caching though, because Dagger automatically caches all actions out of the box :)

Is this strictly because it's using Docker underneath and layers can be reused? If so, unless those intermediary layers are somehow pushed/pulled by the dagger github action (or any associated CI/CD tool equivalent), experience on hosting server is going to be slow.

Sidenote, around 2013 I've worked on a hacky custom container automation workflow within Jenkins for ~100 projects, and spent considerable effort in setting up policies to prune intermediary images.

Thus on certain types of workflows without any prunning a local development machine can be polluted with hundreds of images, unless the user is specifically made aware of stale images. Does/will dagger keep track of the images it builds? I think a command like git gc could make sense.

Re: Dagger: a new way to build CI/CD pipelines

#90

Post by shykes »

cube2222 wrote: post #45

TLDR this basically looks like portable GitHub Actions + Workflows if I understand correctly.

Yes that is a good summary. It has other advantages beyond its portability - but that is the central feature.

Also, importantly, it can run on Github Actions itself!

Post reply on HN