Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jenkins pipelines as YAML

jenkins.io

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Jenkins pipelines as YAML

#31

Post by eddieh »

stefan_ wrote: post #17
ruffrey wrote: post #16

I am so grateful for this. As a non-Java person, using the Groovy pipelines was incredibly painful to incrementally fix on the server. Nothing like pushing 20 commits in a row and forcing a build, only to find each time that an odd syntax error.

Still have to do that to build the thing in the first place. I do not understand at all how every CI solution out there makes it entirely impossible to trivially test a pipeline. Here we are building something whose entire point is that it can run wherever , but the one place it can't run is my local machine? It's like they are doing it out of spite.

I seriously don't understand how anyone can use most of these CI solution.

Here is what everyone should demand from a CI solution:

* Actual code that specifies a pipeline (not a GUI/config-file/DSL)

* Supports any SCM, workflow, build system, and language

* Supports complex mixed-language, mixed-build-system projects

* Doesn't make assumptions about project's structure and processes

* Able to run locally (but scales to clusters, etc)

* Written in a common scripting language

* Completely end-to-end customizable in the scripting language it's written in

The only one I know that fits is BuildBot, but practically nobody uses it.

[edit: formatting]

Re: Jenkins pipelines as YAML

#32

Post by scarface74 »

Chico75 wrote: post #12

This is just lowering themselves to the same level as every other CI tool. For sure YAML is an easier barrier to entry to overcome, but it's also a low ceiling. YAML doesn't let you transcribe logic easily compared to any programming language, and build logic always tends to go up. Programming languages also allows you to create abstraction to remove some of the complexity from the users.

My experience with yml based builds and deployments is basically using the yml file to tell the build system what code to run to do the build - usually other shell scripts or some other scripting language that can also be in the same git repo.

Re: Jenkins pipelines as YAML

#33

Post by scarface74 »

eddieh wrote: post #31
stefan_ wrote: post #17

Earlier quoted context omitted.

Still have to do that to build the thing in the first place. I do not understand at all how every CI solution out there makes it entirely impossible to trivially test a pipeline. Here we are building something whose entire point is that it can run wherever , but the one place it can't run is my local machine? It's like they are doing it out of spite.

I seriously don't understand how anyone can use most of these CI solution. Here is what everyone should demand from a CI solution: * Actual code that specifies a pipeline (not a GUI/config-file/DSL) * Supports any SCM, workflow, build system, and language * Supports complex mixed-language, mixed-build-system projects * Doesn't make assumptions about project's structure and processes * Able to run locally (but scales…

Well if the whole world has a different perspective than you, have you thought that maybe their wants/needs are different?

Re: Jenkins pipelines as YAML

#34

Post by moduspol »

rollulus wrote: post #28

I'm wondering, who uses Jenkins nowadays and why? A few years ago, at one of my first gigs in an enterprise environment, I used Jenkins for the first time to test and build my stuff. When I needed a newer version of my compiler or specific linter, some fellow had to install that on the VM that was running Jenkins. Later, working in a more modern environment, we started using Gitlab CI. It was a bliss. I specified the…

We're using it because it costs nothing and it's what everyone already knows. We're making a big architectural shift to microservices and I made a pitch for the Travis / CircleCI style workflow, but after Jenkins pipelines were discovered, that was the compromise that was made.

We've only got our toes in it now, but from what it looks like, you theoretically can use a Jenkins pipeline (with a Jenkinsfile) to get some of the benefits of those systems, but the problem is that they also allow you to rule out the others. Your Jenkinsfile can assume certain plugins are installed, assume other jobs are configured on the same Jenkins instance... basically all the things that led to Jenkins becoming what it has in terms of a carefully configured sacred cow that must be meticulously backed up and everyone is scared to update.

Having builds trigger on push isn't easy if you're not using GitHub or BitBucket, and having a series of pipelines that trigger off of each other is... not clean. You can certainly trigger another job as a "post" action just like you could in any other Jenkins job, but now your upstream job contains the logic for whether or not a downstream job is triggered. What if your downstream project (like a VM image) only wants builds from a certain branch? Or should hold off on new builds from a certain microservice while QA completes their testing? I guess you'll need to edit the Jenkinsfile for the upstream project (likely someone else's project) and be careful not to break it.

Re: Jenkins pipelines as YAML

#35

Post by manyxcxi »

rollulus wrote: post #28

I'm wondering, who uses Jenkins nowadays and why? A few years ago, at one of my first gigs in an enterprise environment, I used Jenkins for the first time to test and build my stuff. When I needed a newer version of my compiler or specific linter, some fellow had to install that on the VM that was running Jenkins. Later, working in a more modern environment, we started using Gitlab CI. It was a bliss. I specified the…

I’d like to find something friendlier than Jenkins, but the trade offs with the other CIs have made them not worth it or just plain not possible.

If you want or need to host your CI internally, a lot of the shiny new ones are off the table. Additionally, paying per user or per job for our CI is really unpalatable. We host our Jenkins on a few different nodes in our VPC and use them for WAY more than just building and deploying code. Content and database migrations, temporary stack creation and tear down, etc. We have code we build once and ship to multiple environments with different configurations and content, so we have different pipelines that run based off the commit branches, etc. We push to our private Docker, Maven, and npm registries, and auto-deploy via bastion nodes where necessary. On top of that it’s hooked in to our LDAP for auth, which is usually an ‘enterprise’ option that jacks up the price a ridiculous amount.

There’s not much out there that’s mostly batteries included that can handle what we’re doing. Where possible, it would be nice for a team or company to have a Jenkins focused person (back in ye olden days it would’ve been an SCM or build engineer position) because the UI and configuration can definitely be complex.

If you don’t need the complexity, services like BitBucket Pipelines, Travis, DeployBot, and their ilk are certainly much more friendly and likely less error prone. Jenkins definitely still has a valuable place in my book, though.

Re: Jenkins pipelines as YAML

#36

Post by twblalock »

rollulus wrote: post #28

I'm wondering, who uses Jenkins nowadays and why? A few years ago, at one of my first gigs in an enterprise environment, I used Jenkins for the first time to test and build my stuff. When I needed a newer version of my compiler or specific linter, some fellow had to install that on the VM that was running Jenkins. Later, working in a more modern environment, we started using Gitlab CI. It was a bliss. I specified the…

> At my current gig, again an enterprise, it is Jenkins everywhere. They do the most complex things with it, orchestrating entire releases, integration tests, etc.

You just answered your own question -- people use Jenkins because it can do all of those things.

Re: Jenkins pipelines as YAML

#37

Post by nrclark »

rollulus wrote: post #28

I'm wondering, who uses Jenkins nowadays and why? A few years ago, at one of my first gigs in an enterprise environment, I used Jenkins for the first time to test and build my stuff. When I needed a newer version of my compiler or specific linter, some fellow had to install that on the VM that was running Jenkins. Later, working in a more modern environment, we started using Gitlab CI. It was a bliss. I specified the…

Not sure if I can speak for the HN crowd overall. I've never used Gitlab CI, so I can't comment on how I like it. But my experience with Jenkins has been good overall.

Some thoughts:

  - Jenkins is FOSS, which I like.
  - If I want commercial support, I can buy it.
  - Jenkins works effortlessly with Active Directory.
  - Jenkins gives good group-level access control.
  - I don't like the UI very much, but I can live with it.
  - Jenkins is designed to work with remote build agents, which I like a lot.
I think Jenkins' greatest strength is also what some people hate about it: everything is configurable. Its flexibility is a burden if you've got simple build needs. But it's perfect if you've got weird build flows.

I do embedded Linux work. For me, that means I have to deal with weird cross-compilation issues, obscure toolchains, and lots of glue logic between different parts of the build. Jenkins gets the job done better than Travis or Circle, and lets me do it on remote build agents (which is very expensive with Bamboo). I could maybe use Buildbot maybe, but that needs too much customization IMO.

Yes, it's a little crusty. And yes, the UI isn't as pretty as some other CI tools. But it gets the job done with a functional UI and great access control. And you can't beat the price.

Re: Jenkins pipelines as YAML

#38

Post by eddieh »

scarface74 wrote: post #33
eddieh wrote: post #31

Earlier quoted context omitted.

I seriously don't understand how anyone can use most of these CI solution. Here is what everyone should demand from a CI solution: * Actual code that specifies a pipeline (not a GUI/config-file/DSL) * Supports any SCM, workflow, build system, and language * Supports complex mixed-language, mixed-build-system projects * Doesn't make assumptions about project's structure and processes * Able to run locally (but scales…

Well if the whole world has a different perspective than you, have you thought that maybe their wants/needs are different?

Sure that could be the case, but I think most people that set up and use CI just pick Jenkins or whatever on a whim, then we all read about horror stories and bad experiences because the solution they picked is a square peg and they're forcing it in a round hole.

Re: Jenkins pipelines as YAML

#39

Post by humbleMouse »

rhencke wrote: post #24
zdw wrote: post #3

Looks interesting, although it seems very similar to another project, Jenkins Job Builder ( https://docs.openstack.org/infra/jenkins-job-builder/ ) that can describe jobs as YAML or JSON, and is designed one level higher as a meta-jobs creator that can create many jobs all with slight variations, with all of the config kept in version control (no manual fiddling with the Jenkins GUI, other than troubleshooting). With…

Also highly recommended is Jenkins Job DSL, which, despite also using Groovy like Jenkins Pipeline, takes a far different approach. In Job DSL, you write Groovy scripts to declare your job. These scripts, in turn, write the config.xml that Jenkins uses to actually define and execute jobs. One of the big benefits to us is that it is _not_ an alternate execution engine like Jenkins Pipeline is. It just takes over the j…

Agreed, I love the groovy jenkins job DSL. It's so awesome and simple. Easy to put everything on github and forget about it.

Re: Jenkins pipelines as YAML

#40

Post by scarface74 »

eddieh wrote: post #38
scarface74 wrote: post #33

Earlier quoted context omitted.

Well if the whole world has a different perspective than you, have you thought that maybe their wants/needs are different?

Sure that could be the case, but I think most people that set up and use CI just pick Jenkins or whatever on a whim, then we all read about horror stories and bad experiences because the solution they picked is a square peg and they're forcing it in a round hole.

Why do I need all of that bespoked complexity, my yml file is just there to tell the build system the scripts or shell commands I want to run - that are also in my git repo.

Fundamentally, a CI system for a simple application just has to gather dependencies, build a package/run automated tests and store the artifact somewhere.

A deployment system just has to copy the deployment package to a server/group of servers, and do some type of installation process based on an automatic (Dev/integration environment) or manual approval process (who ever is responsible for pushing to any other environment has to approve the release). There are a million ways to skin the cat but if your build process or deployment process is too complex, maybe it’s more of a question of the maturity of your framework or tooling.

My build process for .Net web apps is:

- nuget restore

- msbuild

- run nunit

- zip artifacts based on version.

My deployment process is

- run a CloudFormation file to configure AWS resources

- deploy code to a VM

- run one or two commands to configure IIS or install a Windows service.

- run a few AWS CLI commands to start autoscaling, reconfigure API gateway, etc.

No build servers to maintain, just an agent running on the target VM for deployment and a slightly custom Docker image for builds.

—-

For Lambda functions and scripting languages it’s even simpler...

Build:

- import deprndencies - create zip file

Deployment

- run CloudFormation YML file to deploy lambda

The CF file can be the deployment step. If I need some programmatic, I can create a custom lambda backed resource that is run when the CF file is run.

Post reply on HN