Live data from Hacker News

The Evolution of Container Usage at Netflix

techblog.netflix.com

31–40 of 160 posts

Re: The Evolution of Container Usage at Netflix

#31
> We implemented multi-tenant isolation (CPU, memory, disk, networking and security) using a combination of Linux, Docker and our own isolation technology.

Curious what their 'own isolation technology' does that docker doesn't.

Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this is actually doing.

Re: The Evolution of Container Usage at Netflix

#32
post #28

Earlier quoted context omitted.

It seems they value developer productivity and innovation over machine performance. I find this enlightening.

I value developer productivity over machine performance too, but they're doing the opposite here: developer productivity is not helped with an unnecessary second layer of containment. With containers on top of VMs, you now have to manage which containers run on which VMs (and the cloud provider worries about physical boxes), with a pure container solution you just spawn containers (and the cloud provider worries abou…

It is helped if the second layer further automates deployment. I would imagine Netflix is better at understanding its velocity than we are.

Re: The Evolution of Container Usage at Netflix

#33
post #20
post #15

Netflix Engineering team amazes me a lot. They literally took all the available apis, and build their own platform despite some of the features are already in the AWS offerings. I suppose they did it mainly because the native service isn't flexible and robust enough for their use cases. BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everyt…

They specifically only hire seniors, they don't want to provide positions for juniors as they would prefer to pay more for more seniors than pay indirectly for training juniors. I think this reflects poorly on them, just as much as companies that use OSS but don't contribute anything back.

How does that reflect poorly on them? They are willing to sacrifice potentially great hires because they'd rather make velocity their focal point. It's a trade-off.

Re: The Evolution of Container Usage at Netflix

#34

> We implemented multi-tenant isolation (CPU, memory, disk, networking and security) using a combination of Linux, Docker and our own isolation technology. Curious what their 'own isolation technology' does that docker doesn't. Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this i…

Justifying high engineering salaries, i see this a lot in great companies.

Re: The Evolution of Container Usage at Netflix

#35
post #13
post #2

>The theme that underlies all these improvements is developer innovation velocity I can't wait until this becomes buzzword de jour and startups start using it in their product descriptions. Then someone needs to start talking about products to "enable developer innovation acceleration" to outpace these crufty companies stuck at 25 kph.

Yup. I think further down someone will 'democratize innovation velocity and acceleration'

"unlock innovation potential"

Re: The Evolution of Container Usage at Netflix

#36
post #2

>The theme that underlies all these improvements is developer innovation velocity I can't wait until this becomes buzzword de jour and startups start using it in their product descriptions. Then someone needs to start talking about products to "enable developer innovation acceleration" to outpace these crufty companies stuck at 25 kph.

As long as they include direction and magnitude it's OK with me.

Re: The Evolution of Container Usage at Netflix

#37
post #15

Netflix Engineering team amazes me a lot. They literally took all the available apis, and build their own platform despite some of the features are already in the AWS offerings. I suppose they did it mainly because the native service isn't flexible and robust enough for their use cases. BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everyt…

I asked that in an interview with Netflix, why they were building their own vs open source. The answer I got was "It doesn't work at the scale that Netflix operates" , not quite sure what that means but I didn't press him further.

I work at Cloudflare, since joining almost 3 years ago I've seen multiple pieces of technology get deployed and then replaced because "it just doesn't work at our scale".

It feels kinda weird to say that, but then I see what our DNS servers are doing, or I learn about some customers that struggle to consume their own logs due to the speed at which we produce them, and things that you think are not an area of concern become that... when you have enough traffic flowing through your systems.

Pretty much the only thing that has really stood out is Kafka. Kafka does work at scale.

Re: The Evolution of Container Usage at Netflix

#38
post #34

> We implemented multi-tenant isolation (CPU, memory, disk, networking and security) using a combination of Linux, Docker and our own isolation technology. Curious what their 'own isolation technology' does that docker doesn't. Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this i…

Justifying high engineering salaries, i see this a lot in great companies.

Netflix is the last place I'd expect to do this. They're famous for letting people go when they're no longer needed.

Re: The Evolution of Container Usage at Netflix

#39
post #15

Netflix Engineering team amazes me a lot. They literally took all the available apis, and build their own platform despite some of the features are already in the AWS offerings. I suppose they did it mainly because the native service isn't flexible and robust enough for their use cases. BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everyt…

I asked that in an interview with Netflix, why they were building their own vs open source. The answer I got was "It doesn't work at the scale that Netflix operates" , not quite sure what that means but I didn't press him further.

It's rather rare to see off the shelf technologies (open or closed) that scale to the top 5%. This makes sense because the techniques required at that level are pretty different, and don't apply well to other situations.

So, making an off the shelf product that actually does scale to the top 5% rarely makes sense, since, at best, the customer base will be limited. In the average case, though, the customer base will be almost nil, because of the per-site quirks and customization that are always present.

I experienced this on a weekly basis in the late 1990s and early 2000s at WalMart, where we were centrally managing a network with well over 10 million nodes.

Every vendor ever constantly tried to get in with us, so we had to come up with a triage system or an enormous portion of our time would be spent evaluating.

Even those best of breed would almost never work for us, because of our unprecedented scale (at the time).

In the rare case where an off the shelf product was selected and successful, it had to be heavily customized.

I am of course talking more about 'framework' kinds of tech, or, I guess, things that work at scale. A lot of off the shelf tech was and is used that isn't scale related.

Re: The Evolution of Container Usage at Netflix

#40
> Today, we are in the process of rebuilding how we deploy device-specific server-side logic to our API tier leveraging single core optimized NodeJS servers

Is this the core Netflix API ? have they moved from java - previously, their entire open source contributions were around java (https://netflix.github.io/). Hystrix repo was updated barely a day ago.

For me, this is more interesting than the VM part.

Post reply on HN