Live data from Hacker News

The CD Pipeline Manifesto

manifesto.getglu.dev

41–50 of 51 posts

Re: The CD Pipeline Manifesto

#41
post #21

> Pipeline definitions are scattered across multiple tools—GitHub Actions, Jenkins, ArgoCD, Kubernetes—and environments. This fragmentation leads to confusion, configuration drift, and duplicated effort. So are they talking about some sort of meta language compiling into multiple yaml configs for the different environments or a single separate CI tool that has plugins and integrates with GitHub/gitlab/etc? I do agree…

> So are they talking about … meta language

Said kindly, no, they’re not. They’re just stating values here, imho, not impl detail

Re: The CD Pipeline Manifesto

#42

I had a look at the example glu deployment pipeline and I’m decidedly unimpressed. Admittedly most of my criticism is related to the choice of Go as an implementation language: more than 80% of the code volume is error handling boilerplate! Before the lovers of Go start making the usual arguments consider that in a high-level pipeline script every step is expected to fail in novel and interesting ways! This isn’t “no…

ocamlci is an OCaml Platform offered canned recipe, a la glu, and they really cut that boilerplate down. I almost never use, but i had the same vibes as you did and it made me think of an impl i thought glu may have something to learng from.

https://github.com/ocurrent/ocaml-ci?tab=readme-ov-file

Re: The CD Pipeline Manifesto

#44

> Without types, it is difficult to compose pipelines together. I would gladly hear this argument expanded. It's really not obvious to me that that's the case.

Suppose I give you two functions f, and g. Can you run f(g()) without breaking things? The honest answer is you don't know until you read the functions, which is a slow and difficult thing to do. Suppose I give you functions f and g of respective types int -> str and Nothing -> str. Can you compose them? No, and you see this immediately from the types. Types make reasoning about composability a lot easier. Of course,…

What are side-effects but undocumented arguments and returns?

Firstly, you want to ensure your functions are pure with respect to input. That is to say, they might reference a configuration or context object that is passed to them as an argument, but they'll never reference some global object/variable.

So then the docker image inside some docker registry? Both the image and the registry are values in the config/context argument at the least. Maybe they're their own separate arguments depending on whether you prefer a single big object argument or a bunch of smaller more primitive arguments.

So then the pure function that expects the docker image to exist in some registry is no longer

  Int -> Int
It's now

  String -> String -> Int -> Int
because it needs a registry and an image. Maybe it's

  String -> String -> String -> String -> Int -> Int
because there's a username and password required to access the registry. Icky, but if we make a few types like

  data Registry { 
    user :: String,
    password :: String,
    url :: String
  }
that becomes

  Registry -> String -> Int
But we could make it better by doing something like

  data Foo { 
    reg:: Registry, 
    image :: String
  }
and now the function can be

  Foo -> Int -> Int
This doesn't fix the image not actually existing in the registry, but at least now we know that the two functions aren't composable, and when it fails because the image doesn't exist we can hopefully trace through to see who the caller is that's giving it incorrect data.

PS: sorry if i got the haskell typing wrong. I don't know haskell so that's the result of what i could cobble together from googling about haskell type syntax

Re: The CD Pipeline Manifesto

#45
post #11

I'm thinking before we build a CI/CI pipeline, make sure there is a Makefile. Why have they gone out of style?

I touched make once in 1999, in school. The syntax was arcane, even by 1999 standards.

> Why have they gone out of style?

Because no modern toolchain uses make. Its syntax is so arcane that it's been replaced with various tools that are designed for the specific stack. Otherwise, more generic build systems use modern languages / markup.

Re: The CD Pipeline Manifesto

#46
post #37
post #16

Earlier quoted context omitted.

If I want to build and test a golang app and push it to a container repository, what value does a makefile provide over go build && docker push? All the tools do their own dependency tracking already (unfortunately).

You want to use GNU Make, and then you can ignore the Make dependency tracking. GNU Make is much easier when you only use so-called phony targets (consult the manual), which always execute without doing any dependency tracking. As for the advantage, a makefile will definitely perform both go build and docker push, rather than just (say) docker push, an ever-present risk if you have to rely on your fingers to type the…

fabfile (Fabric/invoke: https://www.pyinvoke.org) may be an alternative to .PHONY-heavy Makefile (shell commands glued together by Python code)

Re: The CD Pipeline Manifesto

#47
post #25
post #24

Earlier quoted context omitted.

Maybe none, but at some point you may want to do other things at buildtime, such as generating an sqlite db or generate code stubs for protobuf. Having a universal, and highly refined, tool like Make will help developers without domain knowledge. It also does not exclude the use of other tool like Just and Docker. A Makefile is also an easy jump-off point for a build pipeline.

Then introduce it for those things. But a makefile to call go build, go test, docker push, Ecs update-services provides no value other than all of a sudden not working on windows without another tool.

I use make on windows under WSL. I've mostly given up on getting everything to run under every OS. There's always some quirk.

Re: The CD Pipeline Manifesto

#48
post #39

Earlier quoted context omitted.

I prefer that, too. I've heard (less-experienced) tech leads forbid Makefiles because they're not declarative enough compared to yaml.

If you stick with what is in common between ninja build and Makefiles, and comment any usage you do of what isn't, the file will mostly mention a series of inputs -> box -> outputs. What happens is make will dispatch in a way that the inputs are all satisfied. It works fine afaict, the only issue is make doesn't contain by itself the tools that it expects to be available in your environment, so you will still need so…

It's quirky, but you can sort of fake dependencies on tools in make too.

Re: The CD Pipeline Manifesto

#49

FTA: > The Fix: Use a full modern programming language, with its existing testing frameworks and tooling. I was reading the article and thinking myself "a lot of this is fixed if the pipeline is just a Python script." And really, if I was to start building a new CI/CD tool today the "user facing" portion would be a Python library that contains helper functions for interfacing with with the larger CI/CD system. Not be…

I don't think I agree. I've now seen the 'language' approach in jenkins and the static yaml file approach in gitlab and drone. A lot of value is to be gained if the whole script can be analysed statically, before execution. E.g. UI Elements can be there and the whole pipeline is visible, before even starting it.

It also serves as a natural sandbox for the "setup" part so we can always know that in a finite (and short) timeline, the script is interpreted and no weird stuff can ever happen.

Of course, there are ways to combine it (e.g. gitlab can generate and then trigger downstream pipelines from within the running CI, but the default is the script. It also has the side effect that pipeline setup can't ever do stuff that cannot be debugged (because it's running _before_ the pipeline) But I concede that this is not that clear-cut. Both have advantages.

Re: The CD Pipeline Manifesto

#50
post #49

FTA: > The Fix: Use a full modern programming language, with its existing testing frameworks and tooling. I was reading the article and thinking myself "a lot of this is fixed if the pipeline is just a Python script." And really, if I was to start building a new CI/CD tool today the "user facing" portion would be a Python library that contains helper functions for interfacing with with the larger CI/CD system. Not be…

I don't think I agree. I've now seen the 'language' approach in jenkins and the static yaml file approach in gitlab and drone. A lot of value is to be gained if the whole script can be analysed statically, before execution. E.g. UI Elements can be there and the whole pipeline is visible, before even starting it. It also serves as a natural sandbox for the "setup" part so we can always know that in a finite (and short…

If you manage to avoid scope creep then sure, static YAML has advantages. But that's not usually what happens, is it? The minute you allow users to execute an outside program -- which is strictly necessary for a CI/CD system -- you've already lost. But even if we ignore that, the number of features always grows over time: you add variables so certain elements can be re-used, then you add loops and conditionals because some things need to happen multiple times, and then you add the ability to do math, string manipulation is always useful, and so on. Before you know it you're trying to solve the halting problem because your "declarative markup" is a poorly specified turing-complete language that just happens to use a YAML parser as a tokenizer. This bespoke language will be strictly worse than Python in every way.

My argument is that we should acknowledge that any CI/CD system intended for wide usage will eventually arrive here, and it's better that we go into that intentionally rather than accidentally.

Post reply on HN