Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Build Code at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

31–40 of 140 posts

Re: How We Build Code at Netflix

#31

Post by matt_wulfeck »

riskable wrote: post #29
chei0aiV wrote: post #18

Earlier quoted context omitted.

> Immutable infrastructure is the future Are there any downsides to immutable infrastructure?

I'm on a team developing an immutable infrastructure solution with (Docker) containers at a large financial institution. The biggest downsides so far: * It can get complicated very quickly since it involves a lot of stages, systems, and loads of brand new tools where you're unlikely to be able to hire existing talent (you need people who can just read the docs on a complicated tool/architecture like Kubernetes and th…

Sounds like Devops.

Re: How We Build Code at Netflix

#32

Post by sytse »

chei0aiV wrote: post #18
sytse wrote: post #16

Earlier quoted context omitted.

Immutable infrastructure is the future and it seems that even Netflix is planning to use containers for that: "Containers provide an interesting potential solution to the last two challenges and we are exploring how containers can help improve our current build, bake, and deploy experience." I think that the future of deployments means that it is closely integrated with your source code management. Every new push bui…

> Immutable infrastructure is the future Are there any downsides to immutable infrastructure?

I think immutable infrastructure still requires a lot of tools and moving parts today. We're spending a lot of time thinking on how to improve this. Two idea's we have:

1. Deploy directly from GitLab to Kubernetes https://gitlab.com/gitlab-org/gitlab-ce/issues/14040 2. Add a container registry to GitLab itself https://gitlab.com/gitlab-org/gitlab-ce/issues/3299

Any comments or suggestions are welcome.

Re: How We Build Code at Netflix

#33

Post by riskable »

dastbe wrote: post #26
riskable wrote: post #20

Earlier quoted context omitted.

The other replies here are great but let me give you the layman's version of what "immutable infrastructure" means: If it works for me it works for everyone. You never patch or upgrade immutable infrastructure. You just replace what you've got with a new VM or container. Containers being preferred because they can be started & stopped near instantaneously and there's nothing like a virtual BIOS that could have differ…

this is why your security team should be integrated into how you (securely!) build and deploy software. in this specific case, that would mean maintaining the OS layer Dockerfile and scanning the dependency tree for build artifacts.

Yeah, it's actually a lot more complicated than that. Let's assume the security team creates the Dockerfile. It'll look like this:

    FROM ubuntu
    WORKDIR ${foo}   # WORKDIR /bar
    RUN developer_script.sh
So now with each new container update you'd need someone from the security team to audit/review `developer_script.sh`. It's kind of pointless if your goal is fast deployments.

If you just need to make sure your developers don't make a mistake in terms of securely configuring their containers (and making sure to always use the latest software) then you simply scan them before the canary stage. The problem there is, "what are you looking for?"

Also, there exists only one tool to scan Docker containers (OpenSCAP) and if your security team doesn't like it, well, you're screwed: https://github.com/OpenSCAP/container-compliance

The other problem is that OpenSCAP only checks the container's packages for compliance. It doesn't actually scan the container's filesystem for things like JREs and bundled libs. So if you're using Docker best practices by keeping your images as minimal as possible you may not even have a package tool inside your containers. In that case how do you check for things like out-of-date versions of Java?

Another problem is you assume there's some modicum of control over what's inside the containers before they're deployed. We're in charge of creating the infrastructure for running containers with the promise that end users (who would be various application teams) can make their own containers (or at least their own Dockerfiles) and deploy them on our infrastructure.

We can work with them to help develop, say, a Kubernetes pod config for their app but as far as what their app is or what gets bundled with it we'll have no knowledge after the first successful deployment.

Re: How We Build Code at Netflix

#34

Post by jdblair »

riskable wrote: post #22
mattiemass wrote: post #2

Very cool article. Amazing how much tooling Netflix has built themselves.

What's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.

Fairly new Netflix employee here (started last November). From what I can see so far the "highly aligned, loosely coupled" culture here works very well. The company has a clear focus, we all get it, and we all work to make it happen. Teams have freedom to accomplish their goals they best way they see fit. There is quite simply not a lot of focus on particular means, the focus is on outcomes.

Re: How We Build Code at Netflix

#35

Post by jfindley »

riskable wrote: post #29
chei0aiV wrote: post #18

Earlier quoted context omitted.

> Immutable infrastructure is the future Are there any downsides to immutable infrastructure?

I'm on a team developing an immutable infrastructure solution with (Docker) containers at a large financial institution. The biggest downsides so far: * It can get complicated very quickly since it involves a lot of stages, systems, and loads of brand new tools where you're unlikely to be able to hire existing talent (you need people who can just read the docs on a complicated tool/architecture like Kubernetes and th…

We are doing something very similar (mesos rather than k8s), also in a financial organisation.

Our main issue is also security - as docker containers aren't very good at actually containing, from a security PoV, we're using SELinux MLS, assigning each container a unique SELinux level/category (fedora-atomic sets this up nicely for you if anyone wants to try).

This is great for security, but breaks so many things - lots of the cutting edge work with docker and related tech isn't done by people that care deeply about security, or really understand this sort of usecase, so we run into a huge amount of edgecases and bugs. We spend a lot of time fighting this.

It also makes having a really good build/deploy pipeline essential, as well as a good environment set up to reproduce and diagnose issues.

Re: How We Build Code at Netflix

#37

Post by draw_down »

riskable wrote: post #22
mattiemass wrote: post #2

Very cool article. Amazing how much tooling Netflix has built themselves.

What's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.

I've been in so many places where that response would be the correct one. I've spent a lot of time decommissioning some rather horrible outcomes of "can't we just build it ourselves".

Re: How We Build Code at Netflix

#38

Post by dastbe »

riskable wrote: post #33
dastbe wrote: post #26

Earlier quoted context omitted.

this is why your security team should be integrated into how you (securely!) build and deploy software. in this specific case, that would mean maintaining the OS layer Dockerfile and scanning the dependency tree for build artifacts.

Yeah, it's actually a lot more complicated than that. Let's assume the security team creates the Dockerfile. It'll look like this: FROM ubuntu WORKDIR ${foo} # WORKDIR /bar RUN developer_script.sh So now with each new container update you'd need someone from the security team to audit/review `developer_script.sh`. It's kind of pointless if your goal is fast deployments. If you just need to make sure your developers d…

I never said you need or should have the package tools inside your container, but you do need to be able to track the lineage of what was put into those containers so you can easily search for ex. deployed container revisions with an out of date Java version. This is where you need a team devoted to build that creates this kind of infrastructure. Otherwise, you end up with difficult to answer questions about your infrastructure.

tl;dr I assume there's some control because there should be.

Re: How We Build Code at Netflix

#39

Post by vlucas »

For those wondering about how this applies to Node.js use a Netflix like myself, it's in there towards the bottom of the article:

> "As Netflix grows and evolves, there is an increasing demand for our build and deploy toolset to provide first-class support for non-JVM languages, like JavaScript/Node.js, Python, Ruby and Go. Our current recommendation for non-JVM applications is to use the Nebula ospackage plugin to produce a Debian package for baking, leaving the build and test pieces to the engineers and the platform’s preferred tooling. While this solves the needs of teams today, we are expanding our tools to be language agnostic."

Re: How We Build Code at Netflix

#40

Post by ArkyBeagle »

riskable wrote: post #22
mattiemass wrote: post #2

Very cool article. Amazing how much tooling Netflix has built themselves.

What's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.

Netflix cares about "it works" and they have to have tools.
Post reply on HN