Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Build Code at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

91–100 of 140 posts

Re: How We Build Code at Netflix

#91

Post by diab0lic »

x0rg wrote: post #67

Isn't Netflix using Mesos (see http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-... )? I don't understand what role it plays here.

The Fenzo scheduler is, if I recall correctly, used to do scheduling for Mantis and Titus (formerly Titan). Think of these as a streaming job infrastructure and a container service respectively. Spinnaker can assist you in deployments to both (I use Spinnaker to deploy Mantis jobs, for example) but the two platforms are not completely prolific -- we still deploy much of the ecosystem as baked AMIs.

Re: How We Build Code at Netflix

#92

Post by toomuchtodo »

seanwilson wrote: post #90
toomuchtodo wrote: post #83

Earlier quoted context omitted.

> That's a false dilemma It's not, and that's coming from someone who does devops for thousands of virtual machines in production. I don't have an hour to burn sometimes to update a repo, build new AMIs, roll them into production, roll the old ones out, watch logs to ensure canaries pass acceptance tests and that production traffic to new instances aren't erring out, all for minor changes (example: nginx mime type ch…

Try something like Heroku. Commit the code to Git, push the repo to the Heroku repo and it deploys it to the servers for you. Easy.

You mean the same Heroku that has the uptime of a server under the dev's desk? No thank you.

Re: How We Build Code at Netflix

#93

Post by vorg »

christop wrote: post #69
mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

Netflix have been quite involved in the Jenkins project, including the Job DSL Plugin, which enables the automated creation of new Jenkins jobs, e.g. when a new Git branch is created, by defining the job structure with a simple Groovy-based DSL. Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-…

For the last 4 months, Groovy has been known as "Apache Groovy".

Re: How We Build Code at Netflix

#95

Post by trollingineer »

vorg wrote: post #93
christop wrote: post #69

Earlier quoted context omitted.

Netflix have been quite involved in the Jenkins project, including the Job DSL Plugin, which enables the automated creation of new Jenkins jobs, e.g. when a new Git branch is created, by defining the job structure with a simple Groovy-based DSL. Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-…

For the last 4 months, Groovy has been known as "Apache Groovy".

Since your birth, you have been known as loser-vorg.

Re: How We Build Code at Netflix

#96

Post by javajosh »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

Is it more reasonable to try to avoid this problem in the first place, or to accept that it's likely to happen and deal with it then? Both options seem reasonable to me (and I'm trying to get my company to adopt Jenkins).

Re: How We Build Code at Netflix

#97

Post by escapee »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

We fought the copy+paste drift for a while. Most jobs were very similar, but just different enough that debugging things when something went wrong was often both time consuming and frustrating.

Ultimately, we took an approach similar to Travis CI, or Gitlab CI [1], only using shell scripts since that plugs into Jenkins easily enough. Every project has a CI script and a release script in a common location relative to the project root that takes care of everything needed to take a fresh clone of a repository, run the tests, and deploy the project (depending on a few environment variables) if the tests pass.

We have 1-click operation to set up a new job in Jenkins, and it handles all the configuration based on an XML template. Everyone understands that they're not supposed to make manual tweaks to the jobs once they're set up, and a year later, things are working pretty smooth.

[1]: http://doc.gitlab.com/ce/ci/quick_start/README.html

Re: How We Build Code at Netflix

#98

Post by yamitty11 »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

> A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software.

Could use something like this if you are using github https://github.com/groupon/DotCi

Job configurations can be version controlled and reviewed like everything else.

Also, the plugin does other optimizations like storing job/build data in a db so jenkins doesn't slow down as you create more builds/jobs. So you don't need "25 Jenkins masters".

For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin

Re: How We Build Code at Netflix

#99

Post by cm2187 »

gjkood wrote: post #94

Major outage being reported worldwide. http://downdetector.com/status/netflix Anything interesting deployed in the last hour? Something in the CI/CD tool chain, Spinnaker, failed for it to move all the way to Live without being caught.

can't connect from the UK...
Post reply on HN