Live data from Hacker News

Dagger Python SDK: Develop Your CI/CD Pipelines as Code

dagger.io

81–90 of 92 posts

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#81
post #46
post #21

Earlier quoted context omitted.

A) Why async in the user code? Is it really necessary? B) Can you mock pipeline events?

> A) Why async in the user code? Is it really necessary? We support both sync and async mode. The Python ecosystem is in a state of flux at the moment between sync and async, so it seemed like the best approach to offer both and let developers choose. > B) Can you mock pipeline events? Could you share a bit more details on what you mean, to make sure I understand correctly?

>We support both sync and async mode. The Python ecosystem is in a state of flux at the moment between sync and async, so it seemed like the best approach to offer both and let developers choose.

Correct me if I'm wrong but it doesn't seem like async will really offer anything a pipeline framework might need?

>Could you share a bit more details on what you mean, to make sure I understand correctly?

I often find myself debugging pipeline complex release workflows that get triggered by a git tag or something.

If I use this thing with github, how can I debug that workflow when it fails?

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#83

This post is for the python sdk, but the golang SDK is fantastic too. Complete gamechanger to be able to move away from jenkins groovy CPS hell

Could you share some information about how you're doing this? How do you create Jenkins stages, push artifacts, test results, etc.? Cheers

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#85
post #21
post #2

Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4

A) Why async in the user code? Is it really necessary? B) Can you mock pipeline events?

> A) Why async in the user code? Is it really necessary?

It's not a requirement, but it's simpler to default to one and mention the other. You can see an example of sync code in https://github.com/helderco/dagger-examples/blob/main/say_sy... and we'll add a guide in the docs website to explain the difference.

Why async?

It's more inclusive. If you want to run dagger from an async environment (say FastAPI), you don't want to run blocking code. You can run the whole pipeline in a thread, but not really taking advantage of the event loop. It's simpler to do the opposite because if you run in a sync environment (like all our examples, running from CLI), it's much easier to just spin an event loop with `anyio.run`.

It's more powerful. For most examples probably the difference is small, unless you're using a lot of async features. Just remove async/await keywords and the event loop. But you can easily reach for concurrency if there's benefit. While the dagger engine ensures most of the parallelism and efficiency, some pipelines can benefit from doing this at the language level. See this example where I'm testing a library (FastAPI) with multiple Python versions: https://github.com/helderco/dagger-examples/blob/main/test_c.... It has an obvious performance benefit compared to running "synchronously": https://github.com/helderco/dagger-examples/blob/main/test_m...

Dagger has a client and a server architecture, so you're sending requests through the network. This is an especially common use case for using async.

Async Python is on the rise. More and more libraries are supporting it, more users are getting to know it, and sometimes it feels very transitional. It's very hard to maintain both async and sync code. There's a lot of duplication because you need blocking and non-blocking versions for a lot of things like network requests, file operations and running subprocesses. But I've made quite an effort to support both and meet you where you're at. I especially took great care to hide the sync/async classes and methods behind common names so it's easy to change from one to another.

I'm very interested to know the community's adoption or preference of one vs the other. :)

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#86
Pipelines as code is the way to go, no matter whether it is CI/CD, data or ML. It’s unfortunate that a lot of pipeline tools use YAML. Glad to see projects like Dagger that use Python instead.

However it is not clear for me what is the benefit of using it instead of calling commands like docker, pytest and kubectl from Python with Plumbum or similar library. Add Fire and it is trivial to create a complete DevOps CLI for your app that can run locally or called from GitHub actions.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#87
post #2

Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4

This is very cool! Also, from the post: > Get started with the Dagger Go SDK, Dagger Python SDK, or let us know which SDK you're looking for. Are you guys seeing many requests for a Dagger Rust SDK yet? :) As someone who writes mostly in Rust, I’d love to get rid of yaml definitions of CI pipelines and instead define pipelines using Rust

Yes, we're getting a bunch of request from rustaceans

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#88

I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…

You can! At least with GitLab. Our pipelines are written in Python, and generate YAML that kick off child-pipelines. It's fairly trivial and works really well. Having for-loops and building an object based on functions making things so much easier.

You start to wonder thy you have to compile to 'yaml', instead of just having gitlab just give you a normale interface, in the form of a library. And then we've come full-circle.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#89
post #2

Hi everyone, Dagger co-founder here. Happy to answer any questions. We also released an update to the Go SDK a few days ago: https://dagger.io/blog/go-sdk-0.4

don't forget about CUE please! Dagger is a good reason to learn CUE, my gut feeling so far is, my life is better with CUE experience.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#90
post #54

Earlier quoted context omitted.

I'd love to learn more! May I ask what your use case, and what you mean by "buildkit access"?

We have some custom thing built on top of jenkins, and we don't get access to a docker socket, so getting dagger to work doesn't seem possible.

how do you use docker in your pipelines then?
Post reply on HN