Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected

gregoryszorc.com

Post reply on HN
Print view View original

111–120 of 184 posts

Re: Modern CI is too complex and misdirected

#111

Post by humanrebar »

pydry wrote: post #101
humanrebar wrote: post #44

Earlier quoted context omitted.

> Build systems inevitably evolve into something turing complete. CI systems are also generally distributed. You want to build and test on all target environments before landing a change or cutting a release! What Turing complete language cleanly models some bits of code running on one environment and then transitions to other code running on an entirely different environment? Folks tend to go declarative to force en…

>What Turing complete language cleanly models some bits of code running on one environment and then transitions to other code running on an entirely different environment? Any language that runs in both environments with an environment abstraction that spans both? >Folks tend to go declarative to force environment-portable configuration. Declarative is always better if you can get away with it. However, it inevitably…

> Any language that runs in both environments with an environment abstraction that spans both?

Do you have examples? This is harder to do than it would seem.

You would need an on demand environment setup (a virtualenv and a lockfile?) or a homogeneous environment and some sort of RPC mechanism (transmit a jar and execute). I expect either to be possible, though I expect the required verbosity and rigor to impede significant adoption.

Basically, I think folks are unrealistic about the ability to be pithy, readable, and robust at the same time.

Re: Modern CI is too complex and misdirected

#112

Post by qbasic_forever »

theptip wrote: post #43
gilbetron wrote: post #13

Earlier quoted context omitted.

Joe Beda (k8s/Heptio) made this same point in one of his TGI Kubernetes videos: https://youtu.be/M_rxPPLG8pU?t=2936 I agree 100%. Every time I see "nindent" in yaml code, a part of my soul turns to dust.

> Every time I see "nindent" in yaml code, a part of my soul turns to dust. Yup. For this reason it's a real shame to me that Helm won and became the lingua franca of composable/configurable k8s manifests. The one benefit of writing in static YAML instead of dynamic , is that regardless of primary programming language, everyone can contribute; more complex systems like KSonnet start exploding in first-use complexity.

I wouldn't say helm has won, honestly. The kubectl tool integerated Kustomize into it and it's sadly way too underutilized. I think it's just that the first wave of k8s tutorials that everyone has learned from were all written when helm was popular. But now with some years of real use people are less keen on helm. There are tons of other good options for config management and templating--I expect to see it keep changing and improving.

Re: Modern CI is too complex and misdirected

#113

Post by kzrdude »

lenkite wrote: post #82

I so oppressed with YAML chosen as the configuration language for mainstream CI systems. How do people manage to live with this ? I always make mistakes - again and again. And I can never keep anything in my head. It's just not natural. Why couldn't they choose a programming language ? Restrict what can be done by all means, but something that has a proper parser, compiler errors and IDE/editor hinting support would…

I guess YAML has the best solution to nested scopes, just indentation

Re: Modern CI is too complex and misdirected

#114

Post by ElijahLynn »

bob1029 wrote: post #33

We got tired of using external tools that were not well-aligned with our build/deployment use cases - non-public network environments. GitHub Actions, et. al. cannot touch the target environments that we deploy our software to. Our customers are also extremely wary of anything cloud-based, so we had to find an approach that would work for everyone. As a result, we have incorporated build & deployment logic into our s…

> GitHub Actions, et. al. cannot touch the target environments that we deploy our software to.

It can with on-prem self-hosted Runners > https://docs.github.com/en/actions/hosting-your-own-runners/...

I just had this same complaint about using Actions and was pointed to this document.

Re: Modern CI is too complex and misdirected

#115

Post by mxcrossb »

karlicoss wrote: post #66
lillecarl wrote: post #23

Earlier quoted context omitted.

What is a well defined "scripting" language? Lua, Python, Ruby? I do agree it'd be nice with a more general purpose language and a lib like you say, but should this lib be implemented in rust/c so that people can easily integrate it into their own language? Many unknowns but great idea.

Literally any real language would be better. Even if I have to learn a bit of it to write pipelines, at least I'll end up with some transferable knowledge as a result. In comparison, if I learned Github Actions syntax, the only thing I know is... Github actions syntax. Useless and isolated knowledge, which doesn't even transfer to other YAML-based systems because each has its own quirks.

On the theme of the posted article “Literally any real language would be better” is how I feel every time I write cmake/auto tools/etc

Re: Modern CI is too complex and misdirected

#116

Post by cosmodisk »

cogman10 wrote: post #87
pbrb wrote: post #39

Earlier quoted context omitted.

I personally love it. RTFM is pretty much the basis of my career. I always at minimum skim the documentation (the entire doc) so I have an index of where to look. It's a great jumping off point for if you do need to google anything. Books are the same. When learning a new language for example, I get a book that covers the language itself (best practices, common design patterns, etc), not how to write a for loop. It s…

lol... yeah.... I've become "the expert" on so much shit just because of RTFM :D It's amazing how much stuff is spelt out in manuals that nobody bothers to read. The only issue is that so few people RTFM that some manuals are pure garbage to try and glean anything useful. In those cases, usually the best route is often to just read the implementation (though that is tedious).

I have someone in my network, who is very active in PHP scene. Tutorials, tips&tricks, code reviews, you name it. He pretty much abandoned his very popular website and went all in on YouTube. Why? Apparently watching video is so much easier than reading 3000 word article.

Re: Modern CI is too complex and misdirected

#117

Post by gryn »

stingraycharles wrote: post #31
jrockway wrote: post #17

I think that modern CI is actually too simple. They all boil down to "get me a Linux box and run a shell script". You can do anything with that, and there are a million different ways to do everything you could possibly want. But, it's easy to implement, and every feature request can be answered with "oh, well just apt-get install foobarbaz3 and run quuxblob to do that." A "too complex" system, would deeply integrate…

I agree with you in principle, but I have learned to accept that this only works for 80% of the functionality. Maybe this works for a simple Diablo or NodeJS project, but in any large production system there is a gray area of “messy shit” you need, and a CI system being able to cater to these problems is a good thing. Dockerizing things is a step in the right direction, at least from the perspective of reproducibilit…

I'm curious what's a Diablo Project ? I've never heard of such technology unless you're speaking of the game with the same name.

Did you possibly mean Django ?

Re: Modern CI is too complex and misdirected

#118

Post by saurik »

jrockway wrote: post #17

I think that modern CI is actually too simple. They all boil down to "get me a Linux box and run a shell script". You can do anything with that, and there are a million different ways to do everything you could possibly want. But, it's easy to implement, and every feature request can be answered with "oh, well just apt-get install foobarbaz3 and run quuxblob to do that." A "too complex" system, would deeply integrate…

I dunno... it seems important to me that CI is as dumb as it can be, because that way it can do anything. If you want that test mechanism you just described, that shouldn't be a property of the CI service, that should be part of the test framework--or testing functionality of the app framework--you use. The tragedy of this super thick service is that CI services suddenly start using your thick integration into your entire development stack as lock-in: right now they are almost fully swappable commodity products... the way it should be.

Re: Modern CI is too complex and misdirected

#119

Post by cyberpunk »

ryan29 wrote: post #83
phtrivier wrote: post #68

Earlier quoted context omitted.

Who are the "people" who really want that, are responsible for a CI build, and are not able to use a full programming language ? I used jenkins pipeline for a while, with groovy scripts. I wish it had been a type checked language to avoid failing a build after 5minutes because of a typo, but, it was working. Then, somehow, the powers that be decided we had to rewrite everything in a declarative pipeline. I still fail…

People used to hate on Gradle a lot, but it was way better than dealing with YAML IMO. Add in the ability to write build scripts in Kotlin and it was looking pretty good before I started doing less Java. I think a CI system using JSON configured via TypeScript would be neat to see. Basically the same thing as Gradle via Kotlin, but for a modern container (ie: Docker) based CI system. I can still go back to Gradle bui…

Uhh, I don't know. All the groovy knobs on Jenkins (especially the cloudbees enterprise one) and nexus enabled a ridiculous amount of customisation which while it made me a load of consultancy money, I think taught me the lesson that most of the time it's better to adapt your apps to your CI and infra, than to try and adapt your CI and infra to your apps.

I much prefer GitLab + k8s to the nightmare of groovy I provided over the last decade anyway..

Re: Modern CI is too complex and misdirected

#120

Post by cyberpunk »

ryan29 wrote: post #109
thinkharderdev wrote: post #102

Earlier quoted context omitted.

It's funny. If you stick around this business long enough you see the same cycles repeated over and over again. When I started in software engineering, builds were done with maven and specified using an XML config. If you had to do anything non-declarative you had to write a plugin (or write a shell script which called separate tasks and had some sort of procedural logic based on the outputs). Then it was gradle (or…

It's so true. I used Ant > Maven > Gradle. The thing that I think is different about modern CI is there's no good, standard way of adding functionality. So it's almost never write a plugin and always hack something together. And none of it's portable between build systems which are (almost) all SaaS, so it's like getting the absolute worst of everything. I'll be absolutely shocked if current CI builds still work in 1…

This is kind of why I like keeping them as dumb as possible. Let each of your repos contain a ./build ./test ./run and the ci does stuff based on those assumptions...

You're switching from rpms->k8s? Actually nothing has to change per repo for this.

Also it creates a nice standard that is easily enforced by your deployment pipelines: no ./run? Then it's undeployable. kthxbye etc..

This becomes important when you have >50 services.

Post reply on HN