Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dagger Python SDK: Develop Your CI/CD Pipelines as Code

dagger.io

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#51

Post by dvasdekis »

Hi guys, love the idea and the code examples look simple and understandable. Nicely done!

One of the reasons we use proprietary pipelines is the automatic 'service principal' login benefits that exist on e.g. Azure Devops, where the pipeline doesn't need to authenticate via secrets or tokens, and instead the running machine has the privileges to interact directly with Azure. (See https://learn.microsoft.com/en-us/azure/devops/pipelines/tas... particularly "addSpnToEnvironment" parameter). I'm sure other clouds have something similar.

Running the same pipeline locally, there are ways to synthetically inject this, but there's no ready support in your framework yet for this (as ideally you'd have an 'authentication' parameter that you can set the details for). Is something like this planned?

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#53

Post by shykes »

dvasdekis wrote: post #51

Hi guys, love the idea and the code examples look simple and understandable. Nicely done! One of the reasons we use proprietary pipelines is the automatic 'service principal' login benefits that exist on e.g. Azure Devops, where the pipeline doesn't need to authenticate via secrets or tokens, and instead the running machine has the privileges to interact directly with Azure. (See https://learn.microsoft.com/en-us/azu…

Thanks for the kind words!

> Running the same pipeline locally, there are ways to synthetically inject this, but there's no ready support in your framework yet for this (as ideally you'd have an 'authentication' parameter that you can set the details for). Is something like this planned?

Yes, we are watching this new pattern of authorizing CI runners very closely. We fully intend to support it, as it seems inevitable that this model will become the standard eventually.

You may actually be able to implement this pattern now, with the current API.

I'm not familiar with this Azure-specific feature, but in the case of OIDC tokens, it's typically as simple as retrieving the ephemeral token from a file or environment variable, and injecting it as an input to your pipeline.

Would you be able to share some more reading material on the "ways to synthetically inject" that you mentioned? We could use that information to devise a plan for supporting it. Also happy to discuss this directly on our Discord server, if you're interested!

Thanks again for the feedback.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#55

Post by qbasic_forever »

mkoubaa wrote: post #8

While I prefer python over yaml, I tend to think imperative build systems are inherently more brittle.

Yeah I would worry without some very strict policies you could slowly end up with your CI taking on insane dependencies like requiring a full numpy and scipy install because someone years ago wanted to make a chart as output. Google went deep into python as a build system and it burned them enough they built Bazel and starlark as a python-like build system but without the rope to hang yourself in full python.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#56

Post by dvasdekis »

shykes wrote: post #53
dvasdekis wrote: post #51

Hi guys, love the idea and the code examples look simple and understandable. Nicely done! One of the reasons we use proprietary pipelines is the automatic 'service principal' login benefits that exist on e.g. Azure Devops, where the pipeline doesn't need to authenticate via secrets or tokens, and instead the running machine has the privileges to interact directly with Azure. (See https://learn.microsoft.com/en-us/azu…

Thanks for the kind words! > Running the same pipeline locally, there are ways to synthetically inject this, but there's no ready support in your framework yet for this (as ideally you'd have an 'authentication' parameter that you can set the details for). Is something like this planned? Yes, we are watching this new pattern of authorizing CI runners very closely. We fully intend to support it, as it seems inevitable…

Logging on to Discord now - I think I can help out with this. Thanks for the welcome! :)

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#57

Post by yakkityyak »

shykes wrote: post #54
yakkityyak wrote: post #52

Wish I could use this for work, but with no buildkit access its a no go :(

I'd love to learn more! May I ask what your use case, and what you mean by "buildkit access"?

We have some custom thing built on top of jenkins, and we don't get access to a docker socket, so getting dagger to work doesn't seem possible.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#58

Post by ngcc_hk »

travisgriggs wrote: post #39

Years ago, I had inherited a complicated C code base with some clever Makefile jujitsu. None of the recent maintainers understood it at its core. They just put a little lipstick here, and a little there to keep things running. I sat down and extracted the compiler/link flags, and then wrote a Python script to do the build. The code was smaller, and built faster. Every “build” engine evolves from being a simple recipe…

Python is good enough.

Maintainability? Just if one start any OO features, one will gradually deal with modern day goto scenario. One can start another round of guessing, especially you have global.

I found it is easier to read a lisp or a c (functional mostly) source than python. But guess we have to use python these days.

Btw, your saying sound like the usual lisp meme - for every … there is a tiny untested lisp engine there.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#59

Post by __MatrixMan__ »

diarrhea wrote: post #20

I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…

I've been using Apache Airflow for CI. It's a square peg in a lot of ways but I like that it's python and I like I can run it locally and iterate by just rerunning a single task instead of the whole pipeline.

Pretty much everything is just a @task decorated python function or a KubernetesPodOperator but sometimes I imagine I'll write CI-focused operators, e.g. terraform.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#60

Post by heydonovan »

diarrhea wrote: post #20

I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…

You can! At least with GitLab. Our pipelines are written in Python, and generate YAML that kick off child-pipelines. It's fairly trivial and works really well. Having for-loops and building an object based on functions making things so much easier.
Post reply on HN