Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dagger Python SDK: Develop Your CI/CD Pipelines as Code

dagger.io

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#61

Post by paulddraper »

heydonovan wrote: post #60
diarrhea wrote: post #20

I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…

You can! At least with GitLab. Our pipelines are written in Python, and generate YAML that kick off child-pipelines. It's fairly trivial and works really well. Having for-loops and building an object based on functions making things so much easier.

Child pipelines work kinda weird.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#63

Post by verdverm »

pydry wrote: post #22
diarrhea wrote: post #20

I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…

>Are Turing-complete languages a bad choice for pipelines? I'd say that non turing complete languages are a bad fit for pipelines. Even mildly complex pipeline will eventually have loops and conditionals. Better python than some originally-a-config-YAML language turned into an imperative monstrosity with loops and conditionals bolted on.

Dagger originally started with CUE, and is still powered by it under the hood, which has the constructs you mention, while also being turing incomplete.

I don't understand this move to define infra and CI imperatively, and tool vendors moving to support umptine languages for their users... Say what the world should look like, not how to get there?

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#64

Post by senorsmile »

chologrande wrote: post #26
courgette wrote: post #24

Earlier quoted context omitted.

Not a snarky question : what about having data and config in yml, and some high level tool like ansible or terraform and the occasional bash? You loop and branch in ansible/terraform I hate that setup but also I have a hard time thinking of something else

Looping in ansible/terraform is exactly the problem. Logic and control flow in YAML/hcl is a nightmare. Plus there's no debugging! You can't set breakpoints in a yaml file or HCL. Adding more YAML to be parsed by other YAML is just terrible at scale.

You definitely can in Ansible.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#65

Post by senorsmile »

shykes wrote: post #49
JackMcMack wrote: post #33

Earlier quoted context omitted.

The api looks very simple and easy, that's a very good sign. We've been doing similar stuff in python, mainly in combination with Pulumi automation. You can do some very fancy things, and I'm sure Dagger would be a nice addition to that. But honestly the thing I'm missing most is a nice UI above all that. How to present realtime and historic logs (and state) for multiple concurrent actions is still not as easy as I t…

> I'm missing most is a nice UI above all that. How to present realtime and historic logs (and state) for multiple concurrent actions is still not as easy as I think it could be. We have plans to solve that :)

What I'd like to see is integrations with existing systems. If I could somehow automatically map stages in dagger to the stages in a Jenkins pipeline, this would be ideal.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#66

Post by senorsmile »

shykes wrote: post #18
aclatuts wrote: post #11

This makes a lot of sense but would it prevent developers from using the default diagrams already built into most CICD websites. I assume migrating to this would make the CICD pipeline look like one step from gitlab, bitbucket, and githubs perspective.

It's up to you how granular you make your CI configuration. Much of it depends on the context and how your team works. If you've already found yourself integrating a Makefile in a CI job, and figuring out the best mapping of Make rules to CI job/step/workflow: this is exactly the same. Ultimately you're just executing a tool which happens to depend on the Dagger engine. How and when you execute it is entirely up to y…

I mentioned this above, but I believe that the killer feature would be integrating with one or more existing front end CI/CD systems so that the dag created by dagger automatically maps to e.g. the pipeline steps of a Jenkins blueocean job.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#67

Post by vvladymyrov »

shykes wrote: post #46
pydry wrote: post #21

Earlier quoted context omitted.

A) Why async in the user code? Is it really necessary? B) Can you mock pipeline events?

> A) Why async in the user code? Is it really necessary? We support both sync and async mode. The Python ecosystem is in a state of flux at the moment between sync and async, so it seemed like the best approach to offer both and let developers choose. > B) Can you mock pipeline events? Could you share a bit more details on what you mean, to make sure I understand correctly?

Related question - as pipelines are not python programs - it would be nice to be able to write unit tests to pipelines code :) And that's where mock pipelines/tasks could be helpful

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#68

Post by shykes »

senorsmile wrote: post #65
shykes wrote: post #49

Earlier quoted context omitted.

> I'm missing most is a nice UI above all that. How to present realtime and historic logs (and state) for multiple concurrent actions is still not as easy as I think it could be. We have plans to solve that :)

What I'd like to see is integrations with existing systems. If I could somehow automatically map stages in dagger to the stages in a Jenkins pipeline, this would be ideal.

I agree, that would be very useful. We have been investigating the best way to deliver this. Probably a generator tool, that would take a dagger project as input, and produce a jenkins configuration as output. This of course could itself be a dagger pipeline :)

Any guidelines on how to structure the final jenkins configuration? What would the stages look like in your project for example? Any mistakes to avoid?

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#69

Post by shykes »

vvladymyrov wrote: post #67
shykes wrote: post #46

Earlier quoted context omitted.

> A) Why async in the user code? Is it really necessary? We support both sync and async mode. The Python ecosystem is in a state of flux at the moment between sync and async, so it seemed like the best approach to offer both and let developers choose. > B) Can you mock pipeline events? Could you share a bit more details on what you mean, to make sure I understand correctly?

Related question - as pipelines are not python programs - it would be nice to be able to write unit tests to pipelines code :) And that's where mock pipelines/tasks could be helpful

You can absolutely test Dagger pipelines, in the same way you test the rest of your code. Just use the dagger library in your tests files, the way you would use any other library. It should work the way you expect.

Re: Dagger Python SDK: Develop Your CI/CD Pipelines as Code

#70

Post by Too »

diarrhea wrote: post #20

I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes. Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itse…

> Are Turing-complete languages a bad choice for pipelines?

Yes and no. If you write “steps” in yaml, you are doing it wrong and might as well be using a Turing complete imperative language.

On the other hand, linear steps isn’t always the best to begin with for a pipeline. Better have a dependency tree such as makefiles but more advanced, that the CI engine can execute in the most optimal order by itself and retry failing steps without restarting from the beginning. Just keep the number of transitions between declarative and Turing complete few, nobody likes templating strings to inject data into a small script.

Post reply on HN