Modern CI is too complex and misdirected • Hacker News Board

Re: Modern CI is too complex and misdirected

#101

Post by pydry » Wed, Apr 07, 2021, 10:16 PM UTC

humanrebar wrote: post #44

pydry wrote: post #11

It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.

> Build systems inevitably evolve into something turing complete. CI systems are also generally distributed. You want to build and test on all target environments before landing a change or cutting a release! What Turing complete language cleanly models some bits of code running on one environment and then transitions to other code running on an entirely different environment? Folks tend to go declarative to force en…

>What Turing complete language cleanly models some bits of code running on one environment and then transitions to other code running on an entirely different environment?

Any language that runs in both environments with an environment abstraction that spans both?

>Folks tend to go declarative to force environment-portable configuration.

Declarative is always better if you can get away with it. However, it inevitably hamstrings what you can do. In most declarative build systems some dirty turing complete hack will inevitably need to be shoehorned in to get the system to do what it's supposed to. A lot of build systems have tried to pretend that this won't happen but it always does eventually once a project grows complex enough.

Re: Modern CI is too complex and misdirected

#102

Post by thinkharderdev » Wed, Apr 07, 2021, 10:19 PM UTC

ryan29 wrote: post #83

phtrivier wrote: post #68

Earlier quoted context omitted.

Who are the "people" who really want that, are responsible for a CI build, and are not able to use a full programming language ? I used jenkins pipeline for a while, with groovy scripts. I wish it had been a type checked language to avoid failing a build after 5minutes because of a typo, but, it was working. Then, somehow, the powers that be decided we had to rewrite everything in a declarative pipeline. I still fail…

People used to hate on Gradle a lot, but it was way better than dealing with YAML IMO. Add in the ability to write build scripts in Kotlin and it was looking pretty good before I started doing less Java. I think a CI system using JSON configured via TypeScript would be neat to see. Basically the same thing as Gradle via Kotlin, but for a modern container (ie: Docker) based CI system. I can still go back to Gradle bui…

It's funny. If you stick around this business long enough you see the same cycles repeated over and over again. When I started in software engineering, builds were done with maven and specified using an XML config. If you had to do anything non-declarative you had to write a plugin (or write a shell script which called separate tasks and had some sort of procedural logic based on the outputs). Then it was gradle (or SBT for me when I started coding mostly in scala) with you could kind of use in a declarative way for simple stuff but also allowed you to just write code for anything custom you needed to do. And one level up you went from Jenkins jobs configured through the UI to Jenkinsfiles. Now I feel like I've come full circle with various GitOps based tools. The build pipeline is now declarative again and for any sort of dynamic behavior you need to either hack something or write a plugin of some sort for the build system which you can invoke in a declarative configuration.

Re: Modern CI is too complex and misdirected

#103

Post by brown9-2 » Wed, Apr 07, 2021, 10:19 PM UTC

jayd16 wrote: post #9

This makes no sense to me. Modern build systems have reproducible results based on strict inputs. Modern CI/CD handles tasks that are not strictly reproducible. The continuous aspect also implies its integrated to source control. I guess I don't understand the post if its not just semantic word games based on sufficient use of the word sufficient. Maybe the point is to talk about how great Taskcluster is but the only…

Your CI pipeline builds and tests your project, which is the same thing your build system does, except they are each using different specifications of how to do that. The author argues this is a waste.

I think by introducing continuous deployment you are changing the topic from what the author wrote (which strictly referred to CI).

Re: Modern CI is too complex and misdirected

#104

Post by viraptor » Wed, Apr 07, 2021, 10:21 PM UTC

cogman10 wrote: post #88

stingraycharles wrote: post #31

Earlier quoted context omitted.

I agree with you in principle, but I have learned to accept that this only works for 80% of the functionality. Maybe this works for a simple Diablo or NodeJS project, but in any large production system there is a gray area of “messy shit” you need, and a CI system being able to cater to these problems is a good thing. Dockerizing things is a step in the right direction, at least from the perspective of reproducibilit…

While it comes up, I think it's more of a rare problem. So much stuff is "x86 linux" or in rare cases "ARM linux" that it doesn't often make sense to have a cross platform CI system. Obviously a db is a counter example. So is node or a compiler. But at least from my experience, a huge number of apps are simply REST/CRUD targeting a homogeneous architecture.

Unless we're talking proprietary software deployed to only one environment, or something really trivial, it's still totally worth testing other environments / architectures.

You'll find dependency compilation issues, path case issues, reserved name usage, assumptions about filesystem layout, etc. which break the code outside of Linux x86.

Re: Modern CI is too complex and misdirected

#105

Post by thinkharderdev » Wed, Apr 07, 2021, 10:25 PM UTC

I've always had a pipe dream of building a CI system on top of something like Airflow. A modern CD pipeline is just a long-running workflow, so it would be great to treat it just like any other data pipeline.

Re: Modern CI is too complex and misdirected

#106

Post by gonzo41 » Wed, Apr 07, 2021, 10:28 PM UTC

thinkharderdev wrote: post #102

ryan29 wrote: post #83

Earlier quoted context omitted.

People used to hate on Gradle a lot, but it was way better than dealing with YAML IMO. Add in the ability to write build scripts in Kotlin and it was looking pretty good before I started doing less Java. I think a CI system using JSON configured via TypeScript would be neat to see. Basically the same thing as Gradle via Kotlin, but for a modern container (ie: Docker) based CI system. I can still go back to Gradle bui…

It's funny. If you stick around this business long enough you see the same cycles repeated over and over again. When I started in software engineering, builds were done with maven and specified using an XML config. If you had to do anything non-declarative you had to write a plugin (or write a shell script which called separate tasks and had some sort of procedural logic based on the outputs). Then it was gradle (or…

Is maven old now....oh uh...gotta get with the cool kids

Re: Modern CI is too complex and misdirected

#107

Post by [deleted] » Wed, Apr 07, 2021, 10:33 PM UTC

cogman10 wrote: post #87

pbrb wrote: post #39

Earlier quoted context omitted.

I personally love it. RTFM is pretty much the basis of my career. I always at minimum skim the documentation (the entire doc) so I have an index of where to look. It's a great jumping off point for if you do need to google anything. Books are the same. When learning a new language for example, I get a book that covers the language itself (best practices, common design patterns, etc), not how to write a for loop. It s…

lol... yeah.... I've become "the expert" on so much shit just because of RTFM :D It's amazing how much stuff is spelt out in manuals that nobody bothers to read. The only issue is that so few people RTFM that some manuals are pure garbage to try and glean anything useful. In those cases, usually the best route is often to just read the implementation (though that is tedious).

[deleted]

Re: Modern CI is too complex and misdirected

#108

Post by qbasic_forever » Wed, Apr 07, 2021, 10:37 PM UTC

pydry wrote: post #98

reubenmorais wrote: post #97

Earlier quoted context omitted.

That's kind of what Bazel does. Skylark is a Python dialect.

Why is an entirely new dialect necessary? Why couldn't it just have been a python library?

It started as that at Google and was a nightmare in the long run. People would sneak in dependencies on non-hermetic or non-reproducible behavior all the time. The classic "this just needs to work, screw it I'm pulling in this library to make it happen" problem. It just kept getting more and more complex to detect and stop those kinds of issues. Hence a new language with no ability to skirt around its hermetic and non-turing nature.

Re: Modern CI is too complex and misdirected

#109

Post by ryan29 » Wed, Apr 07, 2021, 10:41 PM UTC

thinkharderdev wrote: post #102

ryan29 wrote: post #83

Earlier quoted context omitted.

People used to hate on Gradle a lot, but it was way better than dealing with YAML IMO. Add in the ability to write build scripts in Kotlin and it was looking pretty good before I started doing less Java. I think a CI system using JSON configured via TypeScript would be neat to see. Basically the same thing as Gradle via Kotlin, but for a modern container (ie: Docker) based CI system. I can still go back to Gradle bui…

It's funny. If you stick around this business long enough you see the same cycles repeated over and over again. When I started in software engineering, builds were done with maven and specified using an XML config. If you had to do anything non-declarative you had to write a plugin (or write a shell script which called separate tasks and had some sort of procedural logic based on the outputs). Then it was gradle (or…

It's so true. I used Ant > Maven > Gradle. The thing that I think is different about modern CI is there's no good, standard way of adding functionality. So it's almost never write a plugin and always hack something together. And none of it's portable between build systems which are (almost) all SaaS, so it's like getting the absolute worst of everything.

I'll be absolutely shocked if current CI builds still work in 10 years.

Re: Modern CI is too complex and misdirected

#110

Post by qbasic_forever » Wed, Apr 07, 2021, 10:43 PM UTC

Why even have a difference between production and CI in the first place? I see a future where it's Kubernetes all the way down. My CI system is just a different production-like environment. The same job to deploy code to production is used to deploy code to CI. The same deployment that gets a pod running in prod will get a pod running in test under CI. Your prod system handles events from end-users, your CI system handles events from your dev environment and source repo. Everything is consistent and the same. Why should I rely on some bespoke CI system to re-invent everything that my production system already has to do?