Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Untangling Jenkins

conjur.org

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Untangling Jenkins

#41

Post by Matt3o12_ »

grogenaut wrote: post #9
Matt3o12_ wrote: post #7

Earlier quoted context omitted.

I‘m not sure what’s bad about executing arbitrary groovy scripts because you are running arbitrary code anyways (unless Jenkins is not running any code at all – e.g. compile static assets, do something else). If some developer wanted access to the secret files – config files which might have sensitive information such as the database password for the staging environment (I don’t know what they are called in Jenkins)…

The point is that you get mu hich better reuse by not allowing arbitrary groovy and building up interface contracts. This is exactly like any other coding. Using groovy is akin to allowing arbitrary SQL from your web devs intead of apis (from an interface pov not security). Instead just invoke something outside of Jenkins groovy framework like maven, make, etc. Makes you much more portable and upgradable. Jenkins sho…

Got it, my groovy files have always been setup the environment (I.e. write config files for the test database, API keys, etc everything that should not be in git), and then just execute make to make so that I don’t have a completely different setup each environment. Thank you very much!

Re: Untangling Jenkins

#42

Post by harlowja »

RyJones wrote: post #34
grogenaut wrote: post #12

Earlier quoted context omitted.

Can you provide links? I'm on a plane so very hard to Google. Seriously? Downvotes? On a holiday on a plane and I get down votes asking for a little help? I did try and google and didn't find the stuff.

We use JJB, which is YAML based. https://github.com/hyperledger/ci-management/tree/master/jjb for instance

Just fyi :)

Docs:

https://docs.openstack.org/infra/jenkins-job-builder/

Reviews:

https://review.openstack.org/#/q/project:openstack-infra/jen...

Code:

https://git.openstack.org/cgit/openstack-infra/jenkins-job-b...

Re: Untangling Jenkins

#43

Post by oblio »

mikewhy wrote: post #17
oblio wrote: post #2

I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…

Another downside to Jenkins: $200/hr to learn other products do it just as well with much less overhead.

That was actually a joke since I've never done any Jenkins consulting but Jenkins, including the setup cost is waaaay cheaper than other solutions I've seen for what I used it for: distributed cron, queuing system for tasks, higher level dashboard for managers, CI server, etc.

Re: Untangling Jenkins

#44

Post by jonthepirate »

mooreds wrote: post #32
jonthepirate wrote: post #18

I am spearheading an effort to release a CloudFormation template that will launch Jenkins clusters into your AWS account. It represents everything learned while working with Jenkins for over 6 years including 2.5 years of Jenkins at Lyft and 1 year at DoorDash. If you would like to collaborate, please contact block.jon@gmail.com. The project was written with an eye towards making the launching of a Jenkins cluster ve…

I don't have cycles to help, but please post this on HN when you are ready!

Here's a preview of what I'm working on that I just published to YouTube:

https://www.youtube.com/watch?v=GkflqV6KenM

I'd love to hear more about what problems any of you are facing with Jenkins so I can be sure to address them in the work I'm doing.

Jon

Re: Untangling Jenkins

#45

Post by jonthepirate »

BlanketApple wrote: post #5

Man, I'm having this problem right now at work and it's been a nightmare. Admittedly the problem is like 90% cultural. People (on my team) don't understand pipeline and they absolutely don't understand Groovy. Before, they would just write a job with a shell script that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd write one job and apply it to 20 different repos. It worked pretty well. Th…

Rather than run monolithic Jenkins with a kagillion jobs in it that will easily bog down, you could consider running distributed Jenkins in which each team, or even each microservice, gets its own Jenkins Cluster with a small number of slaves and a small number of jobs. See my earlier post in this thread where I posted a YouTube video of the method I use for doing this using a CloudFormation template.

Also, when you say that your developers don't understand groovy, you could teach them how to ding make targets inside of their docker containers. That's the pattern we use at DoorDash... very minimal Jenkins scripting which exists to ding make targets inside Docker images.

Re: Untangling Jenkins

#46

Post by SOLAR_FIELDS »

CodeAlong wrote: post #10
correlation wrote: post #6

I have come up with a slogan for Jenkins: "Jenkins - The butler that does not hesitate to do a disservice". It is currently the most brittle piece of infrastructure we run, with jobs spiralling out of control and slaves crashing with cryptic exceptions. Add to that a horrible plugin ecosystem (tried to install the Slack plugin, it crashed all builds and our webhook endpoint). Add to that groovy (I don't feel like bec…

My workplace has primarily switched all net new ci/cd workflow from Jenkins to Drone ( https://github.com/drone/drone - Drone is a Continuous Delivery system built on container technology.). Having managed masters with 20,000 + configured jobs I agree that Jenkins is a very sore spot for our entire pipeline. We’ve had greater success with increasing the number of masters (one per team or application) and using the ku…

Interesting war story. We have a couple of master instances that go down all the time because of several thousand jobs running. Gonna ask our devops what they are cooking up to resolve and compare to this solution.

Re: Untangling Jenkins

#47

Post by vorg »

grogenaut wrote: post #3
oblio wrote: post #2

I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…

Going off of this I wouldn't even use job dsl or pipelines as they allow you to execute arbitrary groovy. I would consider emulating Travis ci or Amazon pipelines where it is all declarative not imperitive. This way you can re-impliment and consolidate without doing a Turing complete parse of tons of custom groovy scripts

> Travis ci or Amazon pipelines where it is all declarative not imperitive [...] a Turing complete parse of tons of custom groovy scripts

Even worse than Apache Groovy being an imperative DSL (compared to Travis ci and Amazon pipelines being all declarative) is that the most declarative portions of Groovy (i.e. its collections API) are disabled when used in the Jenkins pipelines.

Re: Untangling Jenkins

#48

Post by finspin »

chrisanthropic wrote: post #24

The best Jenkins advice I ever received was from here: https://medium.com/mindera/jenkins-a-la-travis-6c5a8debbb5b On setting up Jenkins Job Builder using a 'seed job'. This setup allows you to add a .jenkins directory to you project, define a yaml based jenkins job inside that directory, push to github, and Jenkins will create the job you defined - NO web UI required. In other words you make Jenkins behave like Trav…

I'm currently looking into how JJB could be used for simple scheduled jobs. But isn't the setup described in the article basically the same what you can do with Jenkinsfile?

Re: Untangling Jenkins

#49

Post by voidr »

grogenaut wrote: post #9
Matt3o12_ wrote: post #7

Earlier quoted context omitted.

I‘m not sure what’s bad about executing arbitrary groovy scripts because you are running arbitrary code anyways (unless Jenkins is not running any code at all – e.g. compile static assets, do something else). If some developer wanted access to the secret files – config files which might have sensitive information such as the database password for the staging environment (I don’t know what they are called in Jenkins)…

The point is that you get mu hich better reuse by not allowing arbitrary groovy and building up interface contracts. This is exactly like any other coding. Using groovy is akin to allowing arbitrary SQL from your web devs intead of apis (from an interface pov not security). Instead just invoke something outside of Jenkins groovy framework like maven, make, etc. Makes you much more portable and upgradable. Jenkins sho…

> Finally doing too much in Jenkins makes it impossible to test Jenkins locally, which means you can neither debug builds locally without pushing new code to build, nor can you test alterations to Jenkins itself locally which makes it much much more fragile.

If you would have used Jenkins pipelines, you would know that you can debug it directly from the UI. I can make my builds hard to run locally on configuration-based systems as well, this is not really an argument against Pipelines.

> How do I know? I'm a sr dev coming in and fixing a mid sized engineering org (400+ devs) jenkins 4 years of accretions.

You haven't used Jenkins Pipelines though, if you would have used it, you would know about the debugging part I mentioned above.

> Build systems should be simple, scalaboe, agile, and composable, not monolithic. Keep the monolithic to the individual builds not cross company.

These have nothing to do with the actual topic, these are just generalisations.

> The point is that you get mu hich better reuse by not allowing arbitrary groovy and building up interface contracts.

You can reuse the groovy code you write, which means you actually get more reuse.

> This is exactly like any other coding. Using groovy is akin to allowing arbitrary SQL from your web devs intead of apis (from an interface pov not security).

That is a bad analogy, a correct one would be something like: writing NodeJS build scripts in JavaScript instead of using Grunt or Gulp, or Groovy vs. Maven/Ant.

> Jenkins should just be your execution/workflow environment not a dev framework.

What is a "dev framework" in your mind?

> Also never mutate the worker boxes, everything should be completely isloated from itself and each other.

What does that have to do with using a programming language instead of a config DSL?

> Jenkins is like any other software architecture.

Fail to see how that vague statement supports your argument.

Re: Untangling Jenkins

#50

Post by voidr »

mikewhy wrote: post #17
oblio wrote: post #2

I've been a build/release engineer for quite a few years now, most spent using Jenkins. It can be quite a heavyweight beast, especially since it's written in Java (so heavy in terms of resource usage) and since it's old (so lots of legacy in terms of architecture). Still, in terms of costs and scalability (up to the level where you have thousands of developers and you probably want to roll your own), I think it is on…

Another downside to Jenkins: $200/hr to learn other products do it just as well with much less overhead.

That's just the poster's consulting price, you can learn everything for free online.
Post reply on HN