Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Container Usage at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

71–80 of 160 posts

Re: The Evolution of Container Usage at Netflix

#71

Post by spydum »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

For sure there is a bit of self-flagellation. However, what it doesn't tell you is how cost effective or as they elude to: how improves developer velocity (a stand in for productivity) is.

Those other guys may also be offering streaming, but how quickly can they adapt?

Re: The Evolution of Container Usage at Netflix

#72

Post by philsnow »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

http://www.smbc-comics.com/comic/2011-12-13

If we make a scale from 1 to 4 out of those four panels, I'd say Google is around a 2.5 and Netflix is somewhere above 3. It's tough to assign numbers much past 3 because the technology becomes indistinguishable from magic pretty quickly.

Re: The Evolution of Container Usage at Netflix

#73

Post by [deleted] »

throwasehasdwi wrote: post #56
djb_hackernews wrote: post #51

Earlier quoted context omitted.

There is a lot of incorrect information in this post. - Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC. - There is no overhead for running processes in containers. - There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host…

I was wrong about docker, back when I was playing with it it did use LXC, and appears to have started out as project to make a specialized version of LXC. You're right that Docker has its own container runtime now. The overhead for running containers is usually very low but real. The OS needs to partition low level resources that are normally shared and the scheduling introduces some overhead. I disagree about networ…

[deleted]

Re: The Evolution of Container Usage at Netflix

#74

Post by psiclops »

throwasehasdwi wrote: post #47
jo909 wrote: post #41

Earlier quoted context omitted.

Could you elaborate on why containers don't perform on EC2? I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is networ…

AFAIK the early go versions of docker used LXC underneath, but that was ~3+ years ago

Re: The Evolution of Container Usage at Netflix

#75

Post by nailer »

sleepybrett wrote: post #59
nailer wrote: post #54

Earlier quoted context omitted.

Late edit: I've said 'Red Shift' above, I mean Red Hat's OpenShift. Tech has too many edgy code names.

.. so kubernetes.

Kubernetes on bare metal, with additional tech to isolate VMs from each other safely. See link elsewhere in thread.

Re: The Evolution of Container Usage at Netflix

#76

Post by nailer »

jhall1468 wrote: post #32
nailer wrote: post #28

Earlier quoted context omitted.

I value developer productivity over machine performance too, but they're doing the opposite here: developer productivity is not helped with an unnecessary second layer of containment. With containers on top of VMs, you now have to manage which containers run on which VMs (and the cloud provider worries about physical boxes), with a pure container solution you just spawn containers (and the cloud provider worries abou…

It is helped if the second layer further automates deployment. I would imagine Netflix is better at understanding its velocity than we are.

Yep but Netflix also has a decade of large scale arch wielded to AWS that we don't.

They might not be deploying containers on VMs on bare metal because they want to, but because they /have/ to.

Re: The Evolution of Container Usage at Netflix

#77

Post by radicalbyte »

1_2__3 wrote: post #42
qeternity wrote: post #18

Earlier quoted context omitted.

Compute is cheap. People are expensive.

The refrain of shitty developers everywhere.

Performance has never mattered. That's why Internet Explorer's map access implementation was exponential instead of logarithmic (to the size of the map).

That one piece of lazy coding held back Javascript on their platform for years, and I'd argue that it as the most significant cause of their browser's market share plummeting (or at last tied with lack of adblockers).

Re: The Evolution of Container Usage at Netflix

#78

Post by throwasehasdwi »

psiclops wrote: post #74
throwasehasdwi wrote: post #47

Earlier quoted context omitted.

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is networ…

AFAIK the early go versions of docker used LXC underneath, but that was ~3+ years ago

Sorry . It's hard to keep up these days, I'm 112 in Javascript years.

Re: The Evolution of Container Usage at Netflix

#79

Post by pm90 »

jo909 wrote: post #41
nailer wrote: post #3

They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing…

Could you elaborate on why containers don't perform on EC2? I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.

I don't know about performance specifically, but you lose a lot of flexibility in managing containers when running them on the public cloud. e.g. there are rate limits so a lot of multicast protocols cannot be used as they quickly saturate that and can't be used effectively (IMO).

Re: The Evolution of Container Usage at Netflix

#80

Post by pm90 »

jo909 wrote: post #58
djb_hackernews wrote: post #51

Earlier quoted context omitted.

There is a lot of incorrect information in this post. - Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC. - There is no overhead for running processes in containers. - There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host…

I agree that the typical NAT or SDN setup around container networking could impact performance or at least require additional resources. But I don't see how that would be any worse on EC2 compared to bare metal or any other hypervisor/provider. Maybe I'm just interpreting too much into the OPs wording and he did not mean that it's a specific EC2 issue.

I've mentioned this in another comment but the short answer is: rate-limiting.

Now, Netflix, being a priority customer, may get higher limits and such. But average joe public cloud user should keep that in mind before trying to use EC2 for running containers.

Post reply on HN