Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4
Any reccs for moving into the IaC space?
Dagger Python SDK: Develop Your CI/CD Pipelines as Code
41–50 of 92 posts
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#42Years ago, I had inherited a complicated C code base with some clever Makefile jujitsu. None of the recent maintainers understood it at its core. They just put a little lipstick here, and a little there to keep things running. I sat down and extracted the compiler/link flags, and then wrote a Python script to do the build. The code was smaller, and built faster. Every “build” engine evolves from being a simple recipe…
> Every “build” engine evolves from being a simple recipe processor
what if I told you it was every machine
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#43Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4
Hey there, this looks interesting. What's the scope of this project? Is it meant to be closer to developers or do you see this being used in production? How is progress of builds, observability, etc being tackled?
> What's the scope of this project? Is it meant to be closer to developers or do you see this being used in production?
Dagger is meant for both development and production. Note that Dagger doesn't run your application itself: only the pipelines to build, test and deploy it. So, although the project is still young and pre-1.0, we expect it will be production-ready more quickly because of the nature of the workloads (running a pipeline is easier than running an app).
> How is progress of builds, observability, etc being tackled?
All Dagger SDKs target the same Dagger Engine. End-users and administrators can target the engine API directly, for logging, instrumentation, etc. The API is not yet publicly documented, but will be soon.
We're also building an optional cloud service, Dagger Cloud, that will provide a lot of these features as a "turnkey" software supply chain management platform.
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#44Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4
It looks awesome! I have a clarification question on this one: > Using the SDK, your program prepares API requests describing pipelines to run, then sends them to the engine. The wire protocol used to communicate with the engine is private and not yet documented, but this will change in the future. For now, the SDK is the only documented API available to your program. Does it mean the sdk is making a round trip to th…
> Does it mean the sdk is making a round trip to the dagger API remotely somewhere, or is the round trip to a locally running docker container?
The short answer, for now, is: "it's complicated" :) There's a detailed explanation of the Dagger Engine architecture here: https://github.com/dagger/dagger/issues/3595
To quote relevant parts:
> The engine is made of 2 parts: an API router, and a runner. > - The router serves API queries and dispatches individual operations to the runner. > - The runner talks to your OCI runtime to execute actual operations. This is basically a buildkit daemon + some glue. > The router currently runs on the client machine, whereas the runner is on a worker machine that will run the containers. This could be the same machine but typically isn’t.
> Eventually we will move the router to a server-side component, tightly coupled and co-located with the runner. This will be shipped as an OCI image which you will be able to provision, administer and upgrade yourself to your heart’s content. This requires non-trivial engineering work, in order to make the API router accessible remotely, and multi-tenant.
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#45I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…
A good first reach is encoding your CI into a first class construct - ideally in the language of your codebase.
CI paths are underdeveloped, which IMO is a huge miss: you pay a developer premium and potentially risk every iteration. Keep the GHA glue light and invest in your code not your lock in.
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#46Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4
A) Why async in the user code? Is it really necessary? B) Can you mock pipeline events?
> A) Why async in the user code? Is it really necessary?
We support both sync and async mode. The Python ecosystem is in a state of flux at the moment between sync and async, so it seemed like the best approach to offer both and let developers choose.
> B) Can you mock pipeline events?
Could you share a bit more details on what you mean, to make sure I understand correctly?
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#47I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…
Think about it from the perspective of the provider of the build platform (e.g. CircleCI, GitLab, GitHub). There are way fewer edge cases parsing YAML files than allowing Turing complete languages.
Be that as it may, YAML has plenty of edge cases that make it the wrong choice.
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#48FYI one of Dagger’s developers runs the most excellent, and breath of fresh air really, Ship It podcast, which focuses on CI/CD and Infrastructure as Code (and always be improving!).
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#49Hey, lead developer for the Python SDK here. I have a few examples in https://github.com/helderco/dagger-examples and I plan to add more. There's also reference documentation in https://dagger-io.readthedocs.io/ so you can get a birds eye view on what's possible.
The api looks very simple and easy, that's a very good sign. We've been doing similar stuff in python, mainly in combination with Pulumi automation. You can do some very fancy things, and I'm sure Dagger would be a nice addition to that. But honestly the thing I'm missing most is a nice UI above all that. How to present realtime and historic logs (and state) for multiple concurrent actions is still not as easy as I t…
> I'm missing most is a nice UI above all that. How to present realtime and historic logs (and state) for multiple concurrent actions is still not as easy as I think it could be.
We have plans to solve that :)
Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code
#50Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4
This is very cool!
Also, from the post:
> Get started with the Dagger Go SDK, Dagger Python SDK, or let us know which SDK you're looking for.
Are you guys seeing many requests for a Dagger Rust SDK yet? :)
As someone who writes mostly in Rust, I’d love to get rid of yaml definitions of CI pipelines and instead define pipelines using Rust