Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dagger: a new way to build CI/CD pipelines

dagger.io

Post reply on HN
Print view View original

241–250 of 267 posts

Re: Dagger: a new way to build CI/CD pipelines

#241

Post by mattste »

shykes wrote: post #63
seeekr wrote: post #58

This seems similar to what https://earthly.dev is doing?

Yes, I believe that is a fair comparison. Earhly is more focused on builds, whereas Dagger has a wider scope: build, test, deployment, any part of a CI/CD pipeline really. But the overall philosophy is the same: run everything in containers. The choice of buildkit as a runtime is also a key similarity. One big difference is that we use CUE as a configuration language, and Earthly uses YAML. We have a lot of respect (…

I recently converted my company's build process to Earthly. I find its syntax to be much easier to grok than CUE. They've also extended/added Docker commands that shore up some of the pain points of working with Dockerfiles.

>Dagger has a wider scope: build, test, deployment, any part of a CI/CD pipeline really

I don't see any reason this can't go into an Earthfile. We have all of these parts in our Earthfiles.

The one common pain point that both Dagger and Earthly haven't solved for me is unifying the machine parallelization with the DAG parallelization. According to this comment[1], it seems like Dagger doesn't have that goal.

For example, we only run our +deploy target if +build, +test and +lint pass. We parallelize each of those targets across workers in Github Actions. I don't know what the solution is to this problem but I know this was annoying to have to handle with Github Action's workflow syntax and horrible to debug locally.

[1] https://news.ycombinator.com/item?id=30859864

Re: Dagger: a new way to build CI/CD pipelines

#242

Post by guelo »

kevinb9n wrote: post #231
TimWillebrands wrote: post #46

Earlier quoted context omitted.

These comments are extremely uninteresting. If you care so much, please just create an issue on their bugtracker or something.

"If you have any questions, I'll be happy to answer them here!"

Except this question.

Re: Dagger: a new way to build CI/CD pipelines

#243

Post by shykes »

loevborg wrote: post #232
chrismatheson wrote: post #214

Earlier quoted context omitted.

I think this comment from the OP answers your Q ? :) https://news.ycombinator.com/item?id=30859125

Thanks for the pointer! I still don't understand 100%. The link states that tasks are parallelized automatically. At the same time, Dagger AIUI is intended to run on top of existing build servers like Jenkins, Circle CI or Github Actions. But then I would assume that some sort of integration between Dagger and the build server needs to be in place so that tasks are parallelized on multiple worker machines (rather tha…

Good question. Let me provide a more detailed answer in terms of concurrency and parallelism. To paraphrase Rob Pike's excellent explanation:

- Concurrency is the breaking down of a program into pieces that can be executed independently.

- Parallelism is the simultaneous execution of multiple things (possibly related, possibly not)

Dagger is designed to be highly concurrent with minimal development effort. Compared to an equivalent configuration in a traditional CI system, your Dagger pipelines will be more concurrent, and require less lines of code.

Because it is highly concurrent, Dagger can be parallelized with relatively little effort. But as you pointed out, you still need to configure parallelization by setting up multiple nodes, etc. Dagger uses buildkit as an excution engine, so parallelizing Dagger boils down to parallelizing buildkit. There is a lot of work in this area (one benefit of building on an existing, mature ecosystem). For example, here is an example of Kubernetes deployment: https://github.com/moby/buildkit/tree/master/examples/kubern...

Note that, because of its highly concurrent design and because of built-in caching, a single node may get you further than you might think. For example, in the e2e testing example: instead of running every test for every new commit, like many CI systems do, Dagger will automatically cache tests with unchanged inputs. So, for example, a change in the documentation will not trigger the API tests; a change in the iOS app source code will not trigger the Android tests; etc.

It's not uncommong for CI runs to become faster after switching to Dagger, without any additional parallelism. As it turns out, most CIs are very wastesful and leave plenty of low-hanging fruits to pick. Dagger helps you pick them :)

Re: Dagger: a new way to build CI/CD pipelines

#244

Post by awinter-py »

shitloadofbooks wrote: post #185
awinter-py wrote: post #183

Earlier quoted context omitted.

IMO the best open source infra lock-in strategy is kube took a while for E/AKS to catch up with G, things like ingress still vary in DX across clouds 'the same everywhere but it only works here' (no comment on morality of this, quality of kube generally, or whether this team will do the same. docker IMO missed the chance to be a cloud host or a standard interface)

Can you elaborate on your concerns? I've not seen them occur in practice at all. Sure each provider does things differently, but they still work with primitive Kubernetes manifests at the end of the day. A migration from one to the other is nothing more than changing some annotations or potentially transforming the "shape" of some lists or maps.

concern: like every system that we think of as cloud software, the 'cloud platform integration' half of it never gets open sourced

same as like, aws extending mysql and postgres to provide RDS features -- they don't open source those pieces, so 'stock oss' DBs don't have fancy RDS features like backup + restore

with kube, platform-integrated components like network, ingress + storage 1) require custom work on each cloud, so new features won't be available on all clouds at the same time.

But also 2) that means some components, ingress specifically, have different interfaces on different clouds when they do finally get implemented.

Re: Dagger: a new way to build CI/CD pipelines

#245

Post by shykes »

mshekow wrote: post #196

How would Dagger compare to Toast? https://github.com/stepchowfun/toast

It seems very similar. The main difference (from a cursory look at their docs) is that toast uses YAML as a frontend, and Docker Engine as a backend. Whereas Dagger uses CUE as the frontend, and Buildkit as a backend.

Re: Dagger: a new way to build CI/CD pipelines

#246

Post by dradtke »

drewm1980 wrote: post #197
shykes wrote: post #4

Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since, you know... Docker. If you have any questions, I'll be happy to answer them here!

The carbon footprint of the cloud has exceeded the footprint of air travel. A move away from monolithic statically compiled binaries to constellations of microservices (usually bloated docker containers) is a significant part of the problem. Docker's explosive growth is partly due to the convenience of the abstraction it provides, abstracting the entire linux userspace, putting even OS-wide package managers and langu…

One of the big points Bill Gates makes in his recent book on climate change is that once the genie is out of the bottle in terms of lifestyle, there's no going back. It's naive to expect people to willingly reduce their energy use enough to make an impact, and immoral when you consider all the people in poor villages that don't even have electricity yet.

The solution is, in a nutshell, to electrify everything, and push to make electricity clean and plentiful. Anything else is doomed to fail because we can't beat climate change by reducing our carbon footprint; we have to eliminate it entirely.

From that perspective, cloud energy usage is not a problem, since it's already electrified by nature. Now we just need to stop emitting carbon in order to make electricity (among other things).

Re: Dagger: a new way to build CI/CD pipelines

#247

Post by littleroot »

shykes wrote: post #4

Hi everyone, I'm one of the co-founders of Dagger (and before that founder of Docker). This is a big day for us, it's our first time sharing something new since, you know... Docker. If you have any questions, I'll be happy to answer them here!

Are you guys aware of Nix, both the language and the build system? Nix at its core is a build system, but the community pushed the boundary of what a "build" means so hard, now Nix could also be used as one definition language for everything in a CI/CD pipeline (also with a canonical collection of "building blocks" in nixpkgs), from (reproducibly) building artifacts to running automated testing/integrating tasks to automatically deliver the artifacts to whatever the "infrastructure" is. After all in a very general sense the whole CI/CD pipeline could be seen as just another build artifact, which I think resonates a lot with your idea. How do you think your project and Nix would overlap and/or (or both) complement each other?

Re: Dagger: a new way to build CI/CD pipelines

#248

Post by nickysielicki »

maxloh wrote: post #204
nickysielicki wrote: post #44

I’m highly jealous of teams that can containerize all their build tools. If you deal with proprietary toolchains that are tens of gigabytes (Windows WDK, Xilinx Vivado/Vitis) it’s just untenable, and that’s before even mentioning licensing. Even Azure doesn’t have a great solution for WDK development. It’s hard to feel like we’re not being left behind. Bind mounting the tools into the container is an option but at th…

I saw a lot of projects build Windows artifacts with GitHub Actions. It doesn't seem too hard.

Okay, here’s an SDK I use. It’s 16GB.

https://docs.microsoft.com/en-us/windows-hardware/drivers/do...

Show me how to use this with GitHub actions, if it’s not too hard.

Re: Dagger: a new way to build CI/CD pipelines

#249

Post by mountainriver »

shykes wrote: post #139
mountainriver wrote: post #112

Earlier quoted context omitted.

I agree everyone knows YAML but its only really great as a config language, not as a general purpose turing-complete programming language. The answer here is more likely to be something like what Pulumi did

Pulumi is great and we think of it as complementary to Dagger: one is focused on managing infrastructure, the other on deploying applications on top of the infrastructure. They are more similar in their choice of language than you might think: * Dagger lets you write actions in any language: Go, Python, Shell, it doesn't matter. Then it runs it in containers, wrapped in a declarative configuration. This allows cross-…

This would be wonderful but isn't what I gather from the docs. I only really see that the pipelines are written in Cue

Re: Dagger: a new way to build CI/CD pipelines

#250

Post by JPKab »

nickysielicki wrote: post #44

I’m highly jealous of teams that can containerize all their build tools. If you deal with proprietary toolchains that are tens of gigabytes (Windows WDK, Xilinx Vivado/Vitis) it’s just untenable, and that’s before even mentioning licensing. Even Azure doesn’t have a great solution for WDK development. It’s hard to feel like we’re not being left behind. Bind mounting the tools into the container is an option but at th…

I've been in your shoes before at a previous gig where we had a highly successful desktop application that ran on Windows. Containerization was an utter nightmare. It's brutal seeing posts like this too, because you feel like the whole world is leaving you behind.

It's made me super grateful to be where I am now, and able to even view tools like this as an option.

Post reply on HN