Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Build Code at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

81–90 of 140 posts

Re: How We Build Code at Netflix

#81

Post by Hovertruck »

markbnj wrote: post #57

I'm a Netflix fan, as a consumer and an engineer, and this blog post just reinforces my fanboi status. Amidst the descriptions of deployment tools and pipelines one thing stood out for me: the fact that AMI bake times are now a large factor, and that "installing packages" and the "snapshotting process" were a big piece of this. Containers are definitely the answer to this problem. You can deploy base images with the…

At Button we switched from baking AMIs to deploying containers on ECS and it's had a pretty amazing effect on deploys, both in terms of the speed and (due to them being quick and painless) frequency.

Re: How We Build Code at Netflix

#82

Post by wtetzner »

seiji wrote: post #23
davis_m wrote: post #14

Earlier quoted context omitted.

It means updating infrastructure by making changes to a versioned service definition instead of on running instances. Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Of course not. You would end up with a running instance of the code that is impossible to replicate. Immutable infrastructure applies that same idea to running services.

Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Absolutely yes in the right circumstances (have you seen how this website works?). Not every service needs to be written as "scale out to a billion nodes" architecture with eight layers of checks and balances between idea to production. We can't globally say "everybody must use a 16 ste…

That's a false dilemma. There are more options than "edit code on running server" and "use a 16-step verifiable app deployment pipeline".

For example: update the code in your source control repository, and then build the new version of the software.

Re: How We Build Code at Netflix

#83

Post by toomuchtodo »

wtetzner wrote: post #82
seiji wrote: post #23

Earlier quoted context omitted.

Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Absolutely yes in the right circumstances (have you seen how this website works?). Not every service needs to be written as "scale out to a billion nodes" architecture with eight layers of checks and balances between idea to production. We can't globally say "everybody must use a 16 ste…

That's a false dilemma. There are more options than "edit code on running server" and "use a 16-step verifiable app deployment pipeline". For example: update the code in your source control repository, and then build the new version of the software.

> That's a false dilemma

It's not, and that's coming from someone who does devops for thousands of virtual machines in production. I don't have an hour to burn sometimes to update a repo, build new AMIs, roll them into production, roll the old ones out, watch logs to ensure canaries pass acceptance tests and that production traffic to new instances aren't erring out, all for minor changes (example: nginx mime type change).

Re: How We Build Code at Netflix

#84

Post by adrianpike »

wtetzner wrote: post #82
seiji wrote: post #23

Earlier quoted context omitted.

Would you fix a software bug by editing the code on a running server and tell yourself that you will add it to the repository later? Absolutely yes in the right circumstances (have you seen how this website works?). Not every service needs to be written as "scale out to a billion nodes" architecture with eight layers of checks and balances between idea to production. We can't globally say "everybody must use a 16 ste…

That's a false dilemma. There are more options than "edit code on running server" and "use a 16-step verifiable app deployment pipeline". For example: update the code in your source control repository, and then build the new version of the software.

Or, since we're talking about adding tooling for better efficiency - just update the source code, and let the tooling build and deploy! :)

Automatic (and safe, obviously) production cutover is still kind of a holy grail, but it's definitely doable, and being done as we speak by a number of pretty neat companies.

Re: How We Build Code at Netflix

#85

Post by oblio »

christop wrote: post #69
mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

Netflix have been quite involved in the Jenkins project, including the Job DSL Plugin, which enables the automated creation of new Jenkins jobs, e.g. when a new Git branch is created, by defining the job structure with a simple Groovy-based DSL. Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-…

Some things are already available, see this: https://github.com/jenkinsci/workflow-plugin/blob/master/REA...

Re: How We Build Code at Netflix

#87

Post by Rafert »

mkobit wrote: post #61

I'm interested in knowing more about the "25 Jenkins masters" that they have, and how much they have modified/built for Jenkins to make it work for them. We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do ev…

I've seen something similar happen, switched to https://www.go.cd/

Re: How We Build Code at Netflix

#89

Post by cyphar »

riskable wrote: post #20
sanjeetsuhag wrote: post #5

Earlier quoted context omitted.

Can some one explain to me what immutable means in terms of "infrastructure" ?

The other replies here are great but let me give you the layman's version of what "immutable infrastructure" means: If it works for me it works for everyone. You never patch or upgrade immutable infrastructure. You just replace what you've got with a new VM or container. Containers being preferred because they can be started & stopped near instantaneously and there's nothing like a virtual BIOS that could have differ…

> So when your security team freaks out because none of your hosts/containers are showing up in their systems you'll have a lot of explaining to do =D

> "We need to scan your hosts so we can ensure that you're installing security patches."

> "We don't do that."

> "You don't install security patches?!?"

> "Yeah, well, you see..."

If you use a tool like zypper-docker, you can create a new image quickly that applies just the security patches. Currently it only works for SUSE-derived containers but we're planning on making it distribution agnostic.

There's also some work we're doing on connecting Docker containers running on SUSE enterprises to connect with SUSE Manager (aka spacewalk), so they would show up in their systems.

You can find most of this stuff in github.com/SUSE.

Re: How We Build Code at Netflix

#90

Post by seanwilson »

toomuchtodo wrote: post #83
wtetzner wrote: post #82

Earlier quoted context omitted.

That's a false dilemma. There are more options than "edit code on running server" and "use a 16-step verifiable app deployment pipeline". For example: update the code in your source control repository, and then build the new version of the software.

> That's a false dilemma It's not, and that's coming from someone who does devops for thousands of virtual machines in production. I don't have an hour to burn sometimes to update a repo, build new AMIs, roll them into production, roll the old ones out, watch logs to ensure canaries pass acceptance tests and that production traffic to new instances aren't erring out, all for minor changes (example: nginx mime type ch…

Try something like Heroku. Commit the code to Git, push the repo to the Heroku repo and it deploys it to the servers for you. Easy.
Post reply on HN