Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shifting Gears

jenkins.io

Post reply on HN
Print view View original

111–120 of 123 posts

Re: Shifting Gears

#111

Post by shinya »

Rowern wrote: post #17
dsumenkovic wrote: post #11

Earlier quoted context omitted.

Glad to hear that. We'd love to hear your feedback about GitLab CI/CD.

I've been working with gitlab CI for the last year. Here are some of my feedbacks: - 6 months ago we seriously considered moving away because it was really unstable (even when running on private runners) but now its a lot smoother - with private runners you can have a very powerful CI without having to manage a master (as Jenkins) for a fraction of the costs (runner with docker-machine on spot instances) - beware tha…

> - IF YOU USE THE BUILT IN CACHE, parallelism will be hard (you cannot populate part of the cache from a job, another part from another job and in the next step use the result of both cache)

You can use the `artifacts` and `dependencies` combo to leverage which artifact will be downloaded into a particular job.

For instance,

    bundle-install:
    stage: build
    script: ...
    artifacts:
        paths: [bin/*]

    yarn-install:
    stage: build
    script: ...
    artifacts:
        paths: [bin/*]

    rspec:
    stage: test
    script: ...
    dependencies: [bundle-install] # This downloads only `bundle-install` artifact to this job

    karma:
    stage: test
    script: ...
    dependencies: [yarn-install] # This downloads only `yarn-install` artifact to this job

    eslint:
    stage: test
    script: ...
    dependencies: [] # This downloads nothing
https://docs.gitlab.com/ee/ci/yaml/#dependencies explains how it works

Re: Shifting Gears

#113

Post by Maxious »

twistadias wrote: post #99
mothsonasloth wrote: post #18

I think this is too much too late for Jenkins. I can't speak for other countries but in London a lot of companies are now using Gitlab or Circle CI. I migrated all my builds (12 projects) to Gitlab CI. After figuring out the first CI pipeline using DockerInDocker, it was easy to then setup the remaining pipelines. Self hosting Gitlab was perfect for our needs (private docker registry). I use Gitlab for personal use t…

I am in London, and our firm uses Jenkins on a huge scale. Certain industries won’t touch products that can’t be installed on premise.

Both GitLab and Circle CI can be installed on premise.

Re: Shifting Gears

#114

Post by shoo »

auxym wrote: post #89
shoo wrote: post #85

Earlier quoted context omitted.

After working in a team that was heavily using Jenkins files & scripted pipelines I started to believe that writing Jenkins scripted pipelines is a bit of an anti pattern, as you end up with lots of build script code that can only run inside of a Jenkins, perhaps coupled to plugins, which hampers your ability to locally develop and test changes. Perhaps sometimes using Jenkins scripted pipeline is a good idea, but if…

I work with Jenkins day in, day out. Doing anything build-script related in Jenkins, whether Pipeline or freestyle jobs, is definitely an anti-pattern. All build-related scripts should definitely be in standalone scripts / build tool config files (make or whatnot), for reasons you describe. Jenkins should be there to handle the "side effects", as I view them. In our case that's stuff like integration with git PRs (po…

It's probably not quite so clear cut. For example, suppose you deploy to AWS, and have automated this to be triggered by a Jenkins job. It's advantageous to be able to run this automated deployment from outside of Jenkins, even though deploying is one real big side effect.

Re: Shifting Gears

#115

Post by ithkuil »

shoo wrote: post #85
nrclark wrote: post #21

I love Jenkins, and use it professionally. With that said, I'd really like to see better documentation on the Jenkinsfile Pipeline format. I've tried to get started with it a few times, and haven't had tons of success. Stuff like "How do I pull in secrets", and "How do I control a plugin". I appreciate that it's Groovy-based, but that's not particularly helpful information (for a hack like me, at least). The snippet-…

After working in a team that was heavily using Jenkins files & scripted pipelines I started to believe that writing Jenkins scripted pipelines is a bit of an anti pattern, as you end up with lots of build script code that can only run inside of a Jenkins, perhaps coupled to plugins, which hampers your ability to locally develop and test changes. Perhaps sometimes using Jenkins scripted pipeline is a good idea, but if…

Where I work it started like this: every "component" had some source in a directory and a couple of simple scripts: build.sh and test.sh.

Then, when we wanted to run them in parallel we just used the parallel Jenkins pipeline statement, so that every step had its own captured output stream (and distinct build statuses too).

This was a slippery slope: now more and more build orchestration complexity moved to groovy code, but fixing that is not obvious because with very long builds seeing which one failed and which not is very useful, and fighting groovy code happens relatively infrequently.

How can I follow your rule of thumb and still let Jenkins capture (possibly in real time) the output and status of work units it doesn't describe and spawn itself?

Re: Shifting Gears

#116

Post by sporkland »

jstrachan wrote: post #40
nrclark wrote: post #21

I love Jenkins, and use it professionally. With that said, I'd really like to see better documentation on the Jenkinsfile Pipeline format. I've tried to get started with it a few times, and haven't had tons of success. Stuff like "How do I pull in secrets", and "How do I control a plugin". I appreciate that it's Groovy-based, but that's not particularly helpful information (for a hack like me, at least). The snippet-…

I think as part of the new Jenkins architecture we should be able to make it much easier to stop at a point in a pipeline & open a terminal/REPL to test out steps. Also I'm hoping for a nice validated YAML based pipeline syntax that should make editing/validating pipelines easier

We have almost the exact opposite request. RE: stop and repl. Our Jenkins pipelines have escalated privs in that they can deploy code so are a juicy attack vector. Wed largely like these things to be read and execute only and any modifications need to go through review.

Re: Shifting Gears

#117

Post by kohsuke »

shoo wrote: post #85
nrclark wrote: post #21

I love Jenkins, and use it professionally. With that said, I'd really like to see better documentation on the Jenkinsfile Pipeline format. I've tried to get started with it a few times, and haven't had tons of success. Stuff like "How do I pull in secrets", and "How do I control a plugin". I appreciate that it's Groovy-based, but that's not particularly helpful information (for a hack like me, at least). The snippet-…

After working in a team that was heavily using Jenkins files & scripted pipelines I started to believe that writing Jenkins scripted pipelines is a bit of an anti pattern, as you end up with lots of build script code that can only run inside of a Jenkins, perhaps coupled to plugins, which hampers your ability to locally develop and test changes. Perhaps sometimes using Jenkins scripted pipeline is a good idea, but if…

With "declarative pipeline" introduced a few years back, this is the direction we are pushing people toward.

Programming capabilities are useful for ecosystem developers to create higher level primitives from existing ones, as it creates a new way of extending Jenkins without plugins.

Re: Shifting Gears

#118

Post by kohsuke »

jonthepirate wrote: post #58

I worked on Jenkins at Lyft and completely set it up for DoorDash. If anybody needs help with their Jenkins setup, hit me up I give free advice and have a few blog posts on the matter. If you happen to be using AWS, GitHub, and Slack, we at DoorDash have developed lots of goodies for streamlining things. We have secured our Jenkins behind our VPN, created load balanced Jenkins clusters, built a shared Groovy library…

You should be presenting in a future Jenkins World event!

Re: Shifting Gears

#119

Post by kohsuke »

chinhodado wrote: post #2

Jenkins's biggest strength is also its biggest weakness: plugins. Any development shop that has been using Jenkins for a while is using at least a bunch of plugins. Plugins are not stable, they break every now and then. They require constant update with new Jenkins versions. They get abandoned by their creators (hell, many plugins still don't support pipeline). It's a fundamental issue with how Jenkins is set up that…

Author of the post. I agree that it is our biggest strength & weakness, I acknowledge that problem and put forward some solutions in the doc.

One piece of the solution is to embrace core and a bunch of important plugins together as the foundation. Normal users shouldn't be asked to pick & choose the basics like that, and we want to lock down the combination of the versions in that group. Whether those are behind the scene plugins or not from contributors' perspective is an implementation detail.

Another piece of the solution is to grow more extensibility mechanism beyond the current in-process plugin. There's a thing called "Pipeline shared libraries" in Jenkins, which is a good example of this. It lets developers create higher level pipeline primitives by composing other existing ones. There's some mechanism to share those with the community, too, although not as sophisticated as plugins. From users' perspective, it extends capabilities of Jenkins just like plugins, but in a way that doesn't create the kind of instability a bad plugin can -- its impact is local to one build, for example.

Then there's the container-as-a-building-block extensibility, Jenkins Evergreen, and more...

Re: Shifting Gears

#120

Post by kohsuke »

rosshemsley wrote: post #69

I was seduced by BlueOcean and tried using Jenkins for CI, using Github and AWS ECS builders (which felt like a common enough use-case). Unfortunately it ended up costing an astonishing amount of engineering time to get working and maintain, with builds frequently stalled or failing. Since moving to CircleCI 2.0 enterprise (admittedly far from perfect) and Airflow, we have _dramatically_ reduced eng. time spent manag…

I'm sorry to hear the bad experience.

I recognize those challenges in my pitch, we have various efforts already under way to address them, and with this gear shifting, I think we'll be combining those in a compelling way.

For example, defining Jenkins config in YAML in Git is a key piece to solve a fear of config change, and this is called "Jenkins Configuration as Code" and is under way for a while now.

Cloud Native Jenkins will also split single process "master" into many build-as-a-function kind of processes, so it isolates builds and allows changes to be rolled out more incrementally.

There's more focus on us owning a bigger responsibilities around "basic every day stuff," too.

Post reply on HN