What is the name for this phenomenon: Observation: X is too complex/too time-consuming/too error-prone. Reaction: Create X' to automate/simplify X. Later: X' is too complex.
"tech"
Modern CI is too complex and misdirected
61–70 of 184 posts
Re: Modern CI is too complex and misdirected
#62Earlier quoted context omitted.
What is a well defined "scripting" language? Lua, Python, Ruby? I do agree it'd be nice with a more general purpose language and a lib like you say, but should this lib be implemented in rust/c so that people can easily integrate it into their own language? Many unknowns but great idea.
I'd say it's not about the capabilities of the language, but the scope of the environment. You need a language to orchestrate your builds and tests (which usually means command execution, variable interpolation, conditional statements and looping constructs), and you need a language to interact with your build system (fetching code, storing and fetching build artifacts, metadata administration). Lua would be a good c…
People hate on it, but do you know what language would be perfect these days?
Easy shelling - check.
Easily embeddable - check
Easily sandboxable - check.
Reasonably rich standard library - check.
High level abstractions - check.
If you're still guessing what language it is, it's Tcl. Good old Tcl.
It's just that is syntax is moderately weird and the documentation available for it is so ancient and creaky that you can sometimes see mummies through the cracks.
Tcl would pretty much solve all these pipeline issues, but it's not a cool language.
I really wish someone with a ton of money and backing would create "TypeTcl" on top of Tcl (à la Typescript and Javascript) and market it to hell and back, create brand new documentation for it, etc.
Re: Modern CI is too complex and misdirected
#63It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
What is a well defined "scripting" language? Lua, Python, Ruby? I do agree it'd be nice with a more general purpose language and a lib like you say, but should this lib be implemented in rust/c so that people can easily integrate it into their own language? Many unknowns but great idea.
Tcl. We already have that language and it's been around for decades, but it's not a cool language. Its community is also ancient, and it feels like it.
Re: Modern CI is too complex and misdirected
#64It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
This is sorta Gradle's approach.
Gradle's approach is commendable, but it's too complicated and they built Gradle on top of Groovy. Groovy is not a good language (and it's also not a good implementation of that not good language).
Re: Modern CI is too complex and misdirected
#65The biggest problem with any CI system is that you need an execution environment. Changing this environment should be the same as changing the code. Docker (or rather podman) has given us the tools to do this.
Now if CI systems would allow me to build that container image myself, I could pretty much guarantee that local build/tests and CI build/tests can run inside the same environment. I hacked something like this for gitlab but it's ugly and slow.
So in conclusion, I think that CI systems should expect container creation, some execution inside that container, and finally some reporting or artifact generation from the results.
Re: Modern CI is too complex and misdirected
#66It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
What is a well defined "scripting" language? Lua, Python, Ruby? I do agree it'd be nice with a more general purpose language and a lib like you say, but should this lib be implemented in rust/c so that people can easily integrate it into their own language? Many unknowns but great idea.
Literally any real language would be better.
Even if I have to learn a bit of it to write pipelines, at least I'll end up with some transferable knowledge as a result.
In comparison, if I learned Github Actions syntax, the only thing I know is... Github actions syntax. Useless and isolated knowledge, which doesn't even transfer to other YAML-based systems because each has its own quirks.
Re: Modern CI is too complex and misdirected
#67I cannot stop thinking this guy is describing what we do at https://garden.io.
He seems to go on describing the Stack Graph and the build/test/deploy/task primitives, the unified configuration between environments, build and test results caching, the platform agnosticism (even though we are heavy focused on kubernetes) and the fact that CI can be just a feature, not a product on itself.
One thing I definitely don't agree with is: "The total addressable market for this idea seems too small for me to see any major player with the technical know-how to implement and offer such a service in the next few years."
We just published the results from an independent survery we commissioned last week and one of the things that came out is: it doesn't matter the size of the company, the amount of hours teams spend mantaining this overly complex build systems, CI systems, Preview/Dev environments etc. is enormous and often is object of the biggest complaints across teams of Tech organizations.
So yeah, I agree with the complexity bit but I think the author is overly positive about the current state of the art, at least in the cloud native world.
Re: Modern CI is too complex and misdirected
#68It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
Scripting languages aren't used directly because people want a declarative format with runtime expansion and pattern matching. We still don't have a great language for that. We just end up embedding snippits in some data format.
Who are the "people" who really want that, are responsible for a CI build, and are not able to use a full programming language ?
I used jenkins pipeline for a while, with groovy scripts. I wish it had been a type checked language to avoid failing a build after 5minutes because of a typo, but, it was working.
Then, somehow, the powers that be decided we had to rewrite everything in a declarative pipeline. I still fail to see the improvement ; but doing "build X, build Y, then if Z build W" is now hard to do.
Re: Modern CI is too complex and misdirected
#69It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
What is a well defined "scripting" language? Lua, Python, Ruby? I do agree it'd be nice with a more general purpose language and a lib like you say, but should this lib be implemented in rust/c so that people can easily integrate it into their own language? Many unknowns but great idea.
Kotlin. It has good support for defining DSLs and can actually type check your pipeline.
Re: Modern CI is too complex and misdirected
#70Isn’t Drone also an example of the author’s ideal solution?