Write Gitlab CI Pipelines in Python Code • Hacker News Board

Re: Write Gitlab CI Pipelines in Python Code

#91

Post by pydry » Fri, Apr 30, 2021, 7:47 PM UTC

infogulch wrote: post #88

pydry wrote: post #84

Earlier quoted context omitted.

The point is you could all get that with YAML without plonking dhall on top if you had a properly designed schema. Hence it's a hack to circumvent badly designed / no schema.

Your argument is akin to "why do you need a high-level language? just write it in assembly, C is just a hack to circumvent badly designed assembly". If a 'properly designed yaml schema' was sufficiently powerful for all these use-cases then why does everyone and their third cousin come up with new configuration languages every week? Clearly they do, thus I must reject the null hypothesis and conclude that plain yaml…

>If a 'properly designed yaml schema' was sufficiently powerful for all these use-cases then why does everyone and their third cousin come up with new configuration languages every week?

Coz they fucked up/didn't fix their schema designs and people keep trying to work around that?

I thought I was being very explicitly obvious about this.

Programming is a history of bad designs being worked around with awkward hacks. This is nothing new.

Re: Write Gitlab CI Pipelines in Python Code

#92

Post by infogulch » Fri, Apr 30, 2021, 7:52 PM UTC

pydry wrote: post #91

infogulch wrote: post #88

Earlier quoted context omitted.

Your argument is akin to "why do you need a high-level language? just write it in assembly, C is just a hack to circumvent badly designed assembly". If a 'properly designed yaml schema' was sufficiently powerful for all these use-cases then why does everyone and their third cousin come up with new configuration languages every week? Clearly they do, thus I must reject the null hypothesis and conclude that plain yaml…

>If a 'properly designed yaml schema' was sufficiently powerful for all these use-cases then why does everyone and their third cousin come up with new configuration languages every week? Coz they fucked up/didn't fix their schema designs and people keep trying to work around that? I thought I was being very explicitly obvious about this. Programming is a history of bad designs being worked around with awkward hacks.…

Ah so your argument is that they just aren't doing it right, that a true yaml schema designer wouldn't make this mistake. It's just a mistake.. which keeps being made over and over by everyone that does configuration. Sounds like a no-true-scotsman and "it's the children who are wrong" to me. Can you give any examples of a popular correctly designed yaml schema?

Re: Write Gitlab CI Pipelines in Python Code

#93

Post by pas » Fri, Apr 30, 2021, 9:37 PM UTC

kortex wrote: post #81

pas wrote: post #55

Earlier quoted context omitted.

Exactly! And this is perfectly achievable with high-level expressive languages too. The important part is that the interface has to have the same semantics as a config file: atomic/transactional. For example the builder pattern does this. You can enjoy all the expressiveness and power of the high-level language, but at some point you create the final config/pipeline/job-graph, call whatever needs to be called, and th…

Yup. My main gripe with Ansible is exactly this. Too much mutable state, too imperative, too much reliance on "gathering facts". There's no singular source of truth. It tries to be declarative, but every Ansible deployment I've maintained (granted, not many) always accretes corner cases and config drift. Maybe that's just the nature of the domain - mutating live systems will always be more imperative than (re)deployi…

... again, exactly. I just use Bash, sacrifice the usual few virgins at certain midnights per year and get on with my life. It's much faster to just type it, easier to debug, no problems with python modules, no YAML.

Sure, it is still Bash, it still requires selling your soul, but it's not like I was going to Haskell heaven anyway.

Re: Write Gitlab CI Pipelines in Python Code

#94

Post by GongOfFour » Sat, May 01, 2021, 12:32 AM UTC

TeeMassive wrote: post #44

GongOfFour wrote: post #38

Earlier quoted context omitted.

In my opinion generating configuration from Python (or any language with a robust standard library) is still a win if it means less Ops and DevOps engineers are stringing together convoluted bash scripts that torture sed/awk/jq/yq to obscene lengths.

While it is true that Python scripts are far more superior to cryptic bash scripts, those scripts shouldn't be on the CI's side. Nothing wrong having build.py and test.py and the others, but there are no good reasons for them to be executable only by the CI. Debugging CI scripts is just asking for non-stop pager duty.

Most all configuration specifications have a limit unless they are turing complete. Complex systems evolve beyond static configurations easily but the question is can you leverage configuration generation to minimize logic inside your CI?

Orchestration ought not be the province of your build scripts, that should be the role of CI. If the configuration isn't capable of variadic functionality but is capable of orchestration... what do you do?

Re: Write Gitlab CI Pipelines in Python Code

#95

Post by pydry » Sun, May 02, 2021, 9:33 PM UTC

infogulch wrote: post #92

pydry wrote: post #91

Earlier quoted context omitted.

>If a 'properly designed yaml schema' was sufficiently powerful for all these use-cases then why does everyone and their third cousin come up with new configuration languages every week? Coz they fucked up/didn't fix their schema designs and people keep trying to work around that? I thought I was being very explicitly obvious about this. Programming is a history of bad designs being worked around with awkward hacks.…

Ah so your argument is that they just aren't doing it right , that a true yaml schema designer wouldn't make this mistake. It's just a mistake.. which keeps being made over and over by everyone that does configuration. Sounds like a no-true-scotsman and "it's the children who are wrong" to me. Can you give any examples of a popular correctly designed yaml schema?

I'm saying that a schema designers reaction to their users using jinja2 or dhall on their YAML ought to be to fix their schema so it's not necessary.

There are plenty of properly designed YAML schemas where people never feel the need to use something like dhall or jinja2.

Re: Write Gitlab CI Pipelines in Python Code

#96

Post by AlphaSite » Sat, May 08, 2021, 10:03 PM UTC

sofixa wrote: post #51

AlphaSite wrote: post #50

Earlier quoted context omitted.

YTT is pretty nice too and basically glues star lark and yaml together.

via comments ! YAML is an abomination in itself ( spaces for logic?!), the last thing it needed was comments for extra logic.

Not actually via comments, they just use the same syntax, im pretty sure the pretemplated code is not actually valid yaml.

Re: Write Gitlab CI Pipelines in Python Code

#97

Post by TeeMassive » Sun, May 09, 2021, 6:01 AM UTC

GongOfFour wrote: post #94

TeeMassive wrote: post #44

Earlier quoted context omitted.

While it is true that Python scripts are far more superior to cryptic bash scripts, those scripts shouldn't be on the CI's side. Nothing wrong having build.py and test.py and the others, but there are no good reasons for them to be executable only by the CI. Debugging CI scripts is just asking for non-stop pager duty.

Most all configuration specifications have a limit unless they are turing complete. Complex systems evolve beyond static configurations easily but the question is can you leverage configuration generation to minimize logic inside your CI? Orchestration ought not be the province of your build scripts, that should be the role of CI. If the configuration isn't capable of variadic functionality but is capable of orchestr…

What is stopping you from making a project dedicated to orchestration/configuration/deployment and using test containers to make those tests? How do you debug those "complex systems" when they fail?

I've made a project in one of my old jobs dedicated to configuration and deployment tools used by CI jobs. At first it got some criticisms about "testing the CI that tests the software" and "testing the tests" but when the number of fires we have to take care of was cut by half it quickly became the norm.