Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Untangling Jenkins

conjur.org

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Untangling Jenkins

#11

Post by ddevault »

If I may suggest my own alternative to Jenkins: builds.sr.ht. It runs build manifests submitted through an API and does not support job configuration itself; you must use an external integration to automate your builds. Build manifests look something like this:

https://git.sr.ht/~sircmpwn/wlroots/tree/.build.yml

More complex example:

https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml

Builds run with full virtualization (KVM) on any number of distributed (optional trust) build runners. The whole thing is open source:

https://git.sr.ht/~sircmpwn/builds.sr.ht

It's also self-hosting, it builds and deploys itself and the rest of the *.sr.ht network:

https://builds.sr.ht/job/602

If anyone wants to give it a try, please get in touch (email in my profile) and I'll hook you up with an account on my hosted instance. I could use more people to kick the wheels and help me find the pain points.

Re: Untangling Jenkins

#12

Post by grogenaut »

oblio wrote: post #4
grogenaut wrote: post #3

Earlier quoted context omitted.

Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts

There's a Declarative Pipeline these days plus you can turn on the Groovy sandbox. There's also a YAML based thing but I forgot its name.

Can you provide links? I'm on a plane so very hard to Google.

Seriously? Downvotes? On a holiday on a plane and I get down votes asking for a little help? I did try and google and didn't find the stuff.

Re: Untangling Jenkins

#13

Post by grogenaut »

BlanketApple wrote: post #5

Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural. People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well. Th…

Why not make it easy to reuse functions to do the steps the groovy is doing and build all of the common steps in a reusable way? Eg what the article is suggesting

Re: Untangling Jenkins

#14

Post by BlanketApple »

grogenaut wrote: post #13
BlanketApple wrote: post #5

Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural. People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well. Th…

Why not make it easy to reuse functions to do the steps the groovy is doing and build all of the common steps in a reusable way? Eg what the article is suggesting

Yeah, we've done that. And again, this is totally a cultural problem.

But inevitably I end up writing that shared groovy code (because I have the most familiarity), and so when things go wrong, or it needs to be changed, I end up being the one having to debug it. Definitely a huge bus factor.

It'd be nice if it was a slightly more common language (e.g. Python) so it wasn't such a pain.

Re: Untangling Jenkins

#15

Post by Everlag »

correlation wrote: post #6

I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice". It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like bec…

Allow me to shill for Concourse[0]. It's oriented around 'tasks' composed into 'pipelines'. Each task either fails or succeeds with the yaml pipeline definition providing hooks into either state to proc other tasks. A task is a yaml definition that invokes a bash script. Each task runs inside a preconfigured docker container for full reproducibility.

It's declarative and everything needed to bring up a pipeline can be stored in git alongside the software it supports. For an example pipeline definition, check out the one for bosh-gcscli[1] and its accompanying tasks[2].

Now, there is the downside that upkeep is a hassle because it requires buying into the BOSH ecosystem to deploy Concourse. That pays for itself by being explicitly anti-brittle, with support for migrations and canary rollouts builtin.

Disclaimer: I worked with Concourse at my previous position but currently have no financial stake in its success.

[0] https://concourse.ci/ [1] https://github.com/cloudfoundry/bosh-gcscli/blob/c908176/ci/... [2] https://github.com/cloudfoundry/bosh-gcscli/tree/30316d6/ci/...

Re: Untangling Jenkins

#16

Post by mikewhy »

correlation wrote: post #6

I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice". It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like bec…

We use Gitlab's built in CI and it's been a dream. Pipelines, Docker, artifacts, caching, coverage, in-repo config, Windows/macOS/Linux.

Re: Untangling Jenkins

#17

Post by mikewhy »

oblio wrote: post #2

I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…

Another downside to Jenkins: $200/hr to learn other products do it just as well with much less overhead.

Re: Untangling Jenkins

#18

Post by jonthepirate »

I am spearheading an effort to release a CloudFormation template that will launch Jenkins clusters into your AWS account. It represents everything learned while working with Jenkins for over 6 years including 2.5 years of Jenkins at Lyft and 1 year at DoorDash. If you would like to collaborate, please contact block.jon@gmail.com. The project was written with an eye towards making the launching of a Jenkins cluster very very easy. It's designed with Docker workloads in mind. Some of the gems include autoscaling, instrumentation & metrics, automatic Let's Encrypt certificate registration, GitHub webhook integration and retries during pipeline slave disconnects which makes spot instances viable even when they abruptly go away... something that saves you 75%!! on your AWS instance costs.

Re: Untangling Jenkins

#19

Post by stephen_g »

correlation wrote: post #6

I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice". It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like bec…

We use JetBrains’ TeamCity CI.

It’s really nice, but expensive if you exceed the limits for the free version (3 agents and 100 build configurations). Luckily we are a fair way off that so can go along fine without having to pay yet.

Post reply on HN