Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Build Code at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: How We Build Code at Netflix

#111

Post by dominotw »

jacques_chester wrote: post #102
Rafert wrote: post #87

Earlier quoted context omitted.

I've seen something similar happen, switched to https://www.go.cd/

GoCD is very difficult to version-control and the interface is, to put it politely, in need of some love. At Pivotal my colleagues working on Cloud Foundry poured a lot of engineering effort into making GoCD scale across multiple teams, repos, sites and so on, and it just never worked out. Alex Suraci wrote Concourse, dogfooded it on a project team, and now pretty much the whole of Cloud Foundry is being built with C…

>now pretty much the whole of Cloud Foundry is being built with Concourse pipelines.

What are some of your thoughts on,

* How it compares to Jenkins pipeline plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin

* Could it have been written a jenkins plugin instead of a whole new CI software. I like some of the features of concourse pipeline but it doesn't have support for wide range of remoting/plugins that jenkins supports.

Re: How We Build Code at Netflix

#112

Post by killface »

escapee wrote: post #97
mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

We fought the copy+paste drift for a while. Most jobs were very similar, but just different enough that debugging things when something went wrong was often both time consuming and frustrating. Ultimately, we took an approach similar to Travis CI, or Gitlab CI [1], only using shell scripts since that plugs into Jenkins easily enough. Every project has a CI script and a release script in a common location relative to…

This way is the only way we've found to do it, even on a smaller scale. Once you get above 20-25 jobs, or create a self service way for teams to create projects+github+jira+cloudformation+jenkins etc, you have to aggressively standardize. Often this means standardizing on the lowest common denominator (shell script).

thanks for sharing.

Re: How We Build Code at Netflix

#113

Post by bmoyles »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles the very basic cases with Jenkins for more complex solutions. Our biggest problems are probably plugin creep (and the poor state of plugin maintenance in some cases), the cluttered UI with chunks of configuration hidden behind "Advanced" buttons, the inability to store job configuration with code a la Travis (which may be better with workflow), and handling routine maintenance across shards (without something in place like Operations Center), API inconsistencies (and the inability to fully manage via the API, forcing things like Groovy scripts over Jenkins remoting...). Once we have some direction on where to go, I'm sure we'll follow up with some more blog posts and share things at our meetups.

Re: How We Build Code at Netflix

#114

Post by jacques_chester »

dominotw wrote: post #111
jacques_chester wrote: post #102

Earlier quoted context omitted.

GoCD is very difficult to version-control and the interface is, to put it politely, in need of some love. At Pivotal my colleagues working on Cloud Foundry poured a lot of engineering effort into making GoCD scale across multiple teams, repos, sites and so on, and it just never worked out. Alex Suraci wrote Concourse, dogfooded it on a project team, and now pretty much the whole of Cloud Foundry is being built with C…

>now pretty much the whole of Cloud Foundry is being built with Concourse pipelines. What are some of your thoughts on, * How it compares to Jenkins pipeline plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin * Could it have been written a jenkins plugin instead of a whole new CI software. I like some of the features of concourse pipeline but it doesn't have support for wide range of remoting/plugins…

I'll defer to the authors for their experiences with Jenkins: http://concourse.ci/concourse-vs.html

As for writing a plugin, no, it would not have been possible. Concourse has an entirely different model of operation and needs easy access to containerisation facilities to achieve it.

Concourse doesn't really think in terms of "plugins". What you become accustomed to is wondering "is there a resource type for this?".

Right now I work on a software repo with a moderately complex API for uploading final binaries and releasing them to clients to download. Instead of telling people to write scripts or install a plugin, I can point them to the resource that another team has written.

"Just add the resource". Released software is now just a stream of events, no different from git commits, S3 files, points in time, Tracker stories etc etc. Every resource has the same interface so it makes it possible to click together stuff into clever combinations, rather than lashing together awkwardly and hoping it'll work.

Re: How We Build Code at Netflix

#115

Post by stakecounter »

riskable wrote: post #22
mattiemass wrote: post #2

Very cool article. Amazing how much tooling Netflix has built themselves.

What's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.

Do they? The only big company I've been at (Amazon) has an absolute monster of an inhouse build and deployment/pipelining system. Although it could just be that they've been around longer than open source alternatives.

Re: How We Build Code at Netflix

#116

Post by jacques_chester »

bmoyles wrote: post #113
mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles…

I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built.

I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way across quite different products (Redis, Apache Geode etc).

People have written resources (the main means of extension) in Go, Ruby and Python so far. If you can put it in a Docker image and execute it, then it can be taught to behave like a Concourse resource.

So far it's working well for us and others trialling it. I am very bullish about the future of Concourse.

Re: How We Build Code at Netflix

#117

Post by relix42 »

gjkood wrote: post #94

Major outage being reported worldwide. http://downdetector.com/status/netflix Anything interesting deployed in the last hour? Something in the CI/CD tool chain, Spinnaker, failed for it to move all the way to Live without being caught.

Not deployment or spinnaker related

Re: How We Build Code at Netflix

#118

Post by igor47 »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

I've spoken to multiple companies about how they do builds, and it's incredible (to me) how many of them use Jenkins. In the end, almost everyone ends up with the kinds of problems you can see on this thread:

  * Plugins are useless
  * Configuration drifts all over the place
  * Jenkins ends up just being used as a job runner to run shell scripts
  * Every repository or project implements similar but subtly different build scripts
When we faced this exact problem while writing Deployboard (https://www.youtube.com/watch?v=tgmJa7FciDg, especially starting at around 18:45), specifically the build system there. I explicitly wanted to avoid having engineers write shell scripts. People don't really know how to write them properly, nobody ever tests them, and they're just not really treated the same way you treat "real code".

Instead, we ended up using resque. We're a rails shop and already use resque extensively in production. Resque scales really really well, so we were pretty confident that we would be able to run builds on this sytem indefinitely without needing to split anything into separate clusters. And resque jobs are ruby, so they can be written and tested just like ruby.

As a result, we were able to standardize on just a few jobs (e.g., BuildRailsJob, GradleBuildJob, NodeBuildJob) with a few arguments for each job. In the process, we wrote a lot of really nice primitives, so if we do need a different kind of job (or a modification to an existing job) then those can be made pretty easily. On the whole, I've been extremely pleased with the resulting system.

Re: How We Build Code at Netflix

#119

Post by geodel »

jacques_chester wrote: post #116
bmoyles wrote: post #113

Earlier quoted context omitted.

We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles…

I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built. I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way acros…

I am going to read documentation of Concourse. Btw have you written like a blog or article about your experience with Concourse?

Re: How We Build Code at Netflix

#120

Post by allengeorge »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

I've seen jenkins devolve into this at one of the previous companies I worked at. There were (at one point) two or three people simply tasked with writing, improving and debugging jenkins plugins as well as random configuration issues and failures. Not a pleasant experience.

Our team ended up forgoing the plugins and simply writing shell scripts (checked into our repository) that would handle various pieces of the build workflow. Our jenkins job then became:

  Call script 1
  Call script 2
  ...
I'm sure there are better alternatives, but at that time it allowed us to version changes to our build, and turned jenkins into nothing more than a glorified task runner - which we were fine with.
Post reply on HN