Untangling Jenkins
conjur.org
Untangling Jenkins
1–10 of 53 posts
Re: Untangling Jenkins
#2I've been a build/release engineer for quite a few years now, most spent using Jenkins.
It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture).
Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is one of the best tools out there for its role as a prettier cron/queuing system.
A few tips:
* if you can afford it, stop ASAP using the UI for configurating jobs; look into Pipelines or the JobDSL, based on your needs
* once you have more than a few jobs, stop running anything on the master node, for increased stability
* use the minimum number of plugins and tie as few things to Jenkins as possible; script as many things and put the scripts into version control
* backup the global configuration; you can try to version control it but it's tricky; at least back it up periodically and most of the time it is enough
That's it for now, for more details, $200/h :p
Re: Untangling Jenkins
#3I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…
Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts
Re: Untangling Jenkins
#4I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…
Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts
There's a Declarative Pipeline these days plus you can turn on the Groovy sandbox. There's also a YAML based thing but I forgot its name.
Re: Untangling Jenkins
#5Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural.
People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well.
That's still an option with pipelines, but it feels much more discouraged (to e.g. write a shared pipeline job that runs a multi-line shell script).
It's also just a bit of an organizational nightmare to see 200+ jobs on the main screen (as opposed to 20 jobs that did the same thing for 200 repos).
Unfortunately, it also seems like declarative pipeline is limited enough that we end up writing a lot of Groovy. I sorta get why Groovy became the official scripting language, but it's like pulling teeth getting people to learn even basic Groovy.
I realize, again, that a lot of this is a cultural problem. Mostly just trying to give a counterpoint that pipelines don't always look as clean as the article implies, especially for orgs with a lot of repos (where committing the same 'test.sh' file to each repo doesn't make a lot of sense).
Re: Untangling Jenkins
#6I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice".
It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like becoming an archeologist).
Gosh, can't someone just build a lightweight ci/cd system ontop of e.g. k8s jobs?
Or should we just give up and do managed ci/cd?
Re: Untangling Jenkins
#7I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…
Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts
I‘m not sure what’s bad about executing arbitrary groovy scripts because you are running arbitrary code anyways (unless Jenkins is not running any code at all – e.g. compile static assets, do something else). If some developer wanted access to the secret files – config files which might have sensitive information such as the database password for the staging environment (I don’t know what they are called in Jenkins) they could just modify the code to dump them to stdout.
I am asking this because I don’t see a problem with running turning compete scripts if the application is most likely to run arbitary code anyways, which has access to syscalls any many other dangerous stuff anyways. You need to trust your employees anyways and if you can’t trust them with not dumping the staging environment in purpose, you should not have hired them. It is possible to include harmless looking code which can be malicious and still be considered a bug if found during code review’s.
Re: Untangling Jenkins
#8I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice". It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like bec…
I replaced it with saltstack. I had so many things not working I thought it was me just being dumb or not spending enough time learning to use it. A few salt states on the end of a git hook did everything, auto deployment, testing, staging, monitoring. I am probably missing something about the awsome power of jenkins but know firms still running my saltstack CI/CD systems years after I left. Weird.
Re: Untangling Jenkins
#9Earlier quoted context omitted.
Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts
I‘m not sure what’s bad about executing arbitrary groovy scripts because you are running arbitrary code anyways (unless Jenkins is not running any code at all – e.g. compile static assets, do something else). If some developer wanted access to the secret files – config files which might have sensitive information such as the database password for the staging environment (I don’t know what they are called in Jenkins)…
The point is that you get mu hich better reuse by not allowing arbitrary groovy and building up interface contracts. This is exactly like any other coding. Using groovy is akin to allowing arbitrary SQL from your web devs intead of apis (from an interface pov not security).
Instead just invoke something outside of Jenkins groovy framework like maven, make, etc. Makes you much more portable and upgradable.
Jenkins should just be your execution/workflow environment not a dev framework.
Also never mutate the worker boxes, everything should be completely isloated from itself and each other.
Jenkins is like any other software architecture.
Finally doing too much in Jenkins makes it impossible to test Jenkins locally, which means you can neither debug builds locally without pushing new code to build, nor can you test alterations to Jenkins itself locally which makes it much much more fragile.
How do I know? I'm a sr dev coming in and fixing a mid sized engineering org (400+ devs) jenkins 4 years of accretions. These are major issues we've identified in just a week of looking.
Build systems should be simple, scalaboe, agile, and composable, not monolithic. Keep the monolithic to the individual builds not cross company.
Re: Untangling Jenkins
#10I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice". It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like bec…
My workplace has primarily switched all net new ci/cd workflow from Jenkins to Drone (https://github.com/drone/drone - Drone is a Continuous Delivery system built on container technology.). Having managed masters with 20,000 + configured jobs I agree that Jenkins is a very sore spot for our entire pipeline. We’ve had greater success with increasing the number of masters (one per team or application) and using the kubernetes plugin for slaves. However, most of our pipeline complaints have been mitigated by getting teams to migrate from Jenkins to Drone.
While I don’t think Drone currently supports k8s job, it is very lightweight in comparison to Jenkins. Most of our developers would agree that it isn’t a perfect product, but it has reduced friction points by a large amount.