Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Build Code at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

71–80 of 140 posts

Re: How We Build Code at Netflix

#71

Post by sytse »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

I think the copy/paste is not inherently bad as long as it is version controlled and visible to everyone, that is why we have all CI configuration in the repo with gitlab-ci.yml

I was surprised about needing 25 different masters. On GitLab.com we have a single clustered application that handles over 1600 GitLab Runners (called build slaves in Jenkins).

Re: How We Build Code at Netflix

#72

Post by onetwotree »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

I think most people do. We did a couple of things to get out of that state at Conjur.

First, we made all of our builds into docker containers. Under this system, slaves (we call 'em executors now) only contain very basic software - docker, git, and make to be specific. This means that our builds are entirely self contained, and we don't have to worry about, for example, messing around with RVM on executors. If the containerized build works locally, it pretty much always works on Jenkins as well.

Second, we started using the job dsl plugin to manage configuration and brought in some autoscaling and machine identity. Beyond what I do as a platform engineer to turn my projects into jenkins builds, I don't completely understand it (nor do I have to, which is a good thing!), so I'll let our DevOps guy take it from here:

https://blog.conjur.net/scaling-jenkins-with-machine-identit...

Re: How We Build Code at Netflix

#73

Post by onetwotree »

sytse wrote: post #71
mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

I think the copy/paste is not inherently bad as long as it is version controlled and visible to everyone, that is why we have all CI configuration in the repo with gitlab-ci.yml I was surprised about needing 25 different masters. On GitLab.com we have a single clustered application that handles over 1600 GitLab Runners (called build slaves in Jenkins).

Agreed, we have a basically arbitrary number of runners (near releases it gets into the hundreds), and one master handles it just fine.

Re: How We Build Code at Netflix

#75

Post by wpietri »

seiji wrote: post #23
davis_m wrote: post #14

Earlier quoted context omitted.

It means updating infrastructure by making changes to a versioned service definition instead of on running instances. Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Of course not. You would end up with a running instance of the code that is impossible to replicate. Immutable infrastructure applies that same idea to running services.

Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Absolutely yes in the right circumstances (have you seen how this website works?). Not every service needs to be written as "scale out to a billion nodes" architecture with eight layers of checks and balances between idea to production. We can't globally say "everybody must use a 16 ste…

I agree with parts of this, sort of.

I maintain a personal server that hosts a variety of VMs for things like mail, static site hosting, nameservice, etc. Last time I rebuilt it, I thought, hey, I'll try doing it with these interesting tools. The immutable-infrastructure approach turns out to be pretty heavy in that case. A lot of times it feels like I have to perform a triple bank shot just to make a minor configuration change.

That said, I think the one-off server, carefully maintained by one person, is becoming very rare. When I build things that other people will work on, I think immutable infrastructure and the cattle-not-pets approach is the only responsible way to go. Even if traffic volume won't be huge, I think the clarity and ease of debugging you get is vital when it doesn't all live in one wizard's head.

I agree that stateful servers are a challenge in this context. But they're a challenge regardless. Having that one thing we're afraid to upgrade or restart can cause all sorts of development and business process issues. That might have been a worthwhile tradeoff 10 years ago when you had to buy all your hardware and you needed ops people to do a lot of stuff manually. Servers go down, and we might as well accept that from the beginning.

Re: How We Build Code at Netflix

#78

Post by Gratsby »

> The Netflix culture of freedom and responsibility empowers engineers to craft solutions using whatever tools they feel are best suited to the task.

I absolutely love that. I'm a huge fan of what Hastings and company have done over there in terms of culture and making Netflix a unique and desirable place to work.

I think it's time for another round of "find a way to make Netflix hire me."

Re: How We Build Code at Netflix

#79

Post by draw_down »

riskable wrote: post #53
draw_down wrote: post #37

Earlier quoted context omitted.

I've been in so many places where that response would be the correct one. I've spent a lot of time decommissioning some rather horrible outcomes of "can't we just build it ourselves".

When companies treat engineers like disposable commodities they should expect solutions that will be disposed.

NIH is the opposite of "disposable engineers".

It's saying your engineers are so much more precious and able than the barbarians outside the gate, that they should write their own test harness/DOM selection library/insert commodity software genre here.

Re: How We Build Code at Netflix

#80

Post by sytse »

onetwotree wrote: post #73
sytse wrote: post #71

Earlier quoted context omitted.

I think the copy/paste is not inherently bad as long as it is version controlled and visible to everyone, that is why we have all CI configuration in the repo with gitlab-ci.yml I was surprised about needing 25 different masters. On GitLab.com we have a single clustered application that handles over 1600 GitLab Runners (called build slaves in Jenkins).

Agreed, we have a basically arbitrary number of runners (near releases it gets into the hundreds), and one master handles it just fine.

Glad to hear that. We'll soon announce an autoscaling runner that allows you to boot up new instances automatically. I love that you sometimes run hundreds of runners, would you like to do a guest post? Email me at website@sytse.com
Post reply on HN