Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Container Usage at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

51–60 of 160 posts

Re: The Evolution of Container Usage at Netflix

#51

Post by djb_hackernews »

throwasehasdwi wrote: post #47
jo909 wrote: post #41

Earlier quoted context omitted.

Could you elaborate on why containers don't perform on EC2? I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is networ…

There is a lot of incorrect information in this post.

- Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC.

- There is no overhead for running processes in containers.

- There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host's network capabilities. Otherwise it is typically a combination of bridges and overlay networks for which the benefits outweigh the performance concerns for most workloads.

Re: The Evolution of Container Usage at Netflix

#52

Post by RcouF1uZ4gsC »

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box.

In regard to the analytics, they have 100 million subscribers. Let's say each subscriber watches an average of 100 episodes/movies a day. For each watch you record subscriber ID, movie ID, start time, stop time and get 32 bytes * 100 * 100,000,000 = 320 gigabytes of data per day total. I am pretty sure that you could get a commercial database and and business intelligence package that could support the type of analytics you need (mainly clustering analysis) at that scale. A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need. In addition, I have subscribed to multiple Netflix type services and I have never weighed the quality of suggestions very high, giving much more weight to the functionality of the client, lack of ads, and large catalog of good movies.

As evidence that this is a solved engineering problem, just look at the number of similar movie services: Amazon, Google play, Hulu, PlayStation Vue, Pureflix, Crackle, etc.

Google, Facebook, Baidu, Amazon, the self driving car companies are doing cutting edge stuff in terms of scalability and analysis, but not Netflix. The complexity of their operations, seems mainly to be one of their own doing and not intrinsic to the service they provide.

So I look at stuff like the article here and see a bunch of very smart engineers who are bored with the (solved) core problem and spend their time making cool stuff which is actually a pretty good thing.

Re: The Evolution of Container Usage at Netflix

#53

Post by nailer »

lima wrote: post #45
nailer wrote: post #3

They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing…

SELinux is NOT isolation. The main issue is the large kernel attack surface and SELinux, while important, only solves a small part of that.

That's true, but a lot of the classic containers-not-containing issues (/sysfs hacks to get into the parent kernel etc) are prevented by SELinux policies.

See https://blog.openshift.com/securing-dockers-future-with-seli...

Re: The Evolution of Container Usage at Netflix

#54

Post by nailer »

nailer wrote: post #3

They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing…

Late edit: I've said 'Red Shift' above, I mean Red Hat's OpenShift.

Tech has too many edgy code names.

Re: The Evolution of Container Usage at Netflix

#55

Post by [deleted] »

nailer wrote: post #3

They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing…

[deleted]

Re: The Evolution of Container Usage at Netflix

#56

Post by throwasehasdwi »

djb_hackernews wrote: post #51
throwasehasdwi wrote: post #47

Earlier quoted context omitted.

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is networ…

There is a lot of incorrect information in this post. - Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC. - There is no overhead for running processes in containers. - There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host…

I was wrong about docker, back when I was playing with it it did use LXC, and appears to have started out as project to make a specialized version of LXC. You're right that Docker has its own container runtime now.

The overhead for running containers is usually very low but real. The OS needs to partition low level resources that are normally shared and the scheduling introduces some overhead.

I disagree about network performance. The virtualization adds a somewhat small but non-trivial overhead here (the overhead for other stuff could probably be considered trivial)

Here is a paper I dug up on that gives results to back up my ranting. It's a bit old now but probably still holds mostly true. http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...

Re: The Evolution of Container Usage at Netflix

#57

Post by brianwawok »

scurvy wrote: post #44
brianwawok wrote: post #19

How does this compare to k8s? Seems like it has everything plus a layer of scheduling and batch jobs on top?

Doesn't kubernetes run an overlay network (at least in some scenarios)?

That is true, it has more fancy network stuff, but less scheduling stuff.

This space is getting busy in the last few years..

Re: The Evolution of Container Usage at Netflix

#58

Post by jo909 »

djb_hackernews wrote: post #51
throwasehasdwi wrote: post #47

Earlier quoted context omitted.

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is networ…

There is a lot of incorrect information in this post. - Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC. - There is no overhead for running processes in containers. - There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host…

I agree that the typical NAT or SDN setup around container networking could impact performance or at least require additional resources.

But I don't see how that would be any worse on EC2 compared to bare metal or any other hypervisor/provider.

Maybe I'm just interpreting too much into the OPs wording and he did not mean that it's a specific EC2 issue.

Re: The Evolution of Container Usage at Netflix

#59

Post by sleepybrett »

nailer wrote: post #54
nailer wrote: post #3

They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing…

Late edit: I've said 'Red Shift' above, I mean Red Hat's OpenShift. Tech has too many edgy code names.

.. so kubernetes.

Re: The Evolution of Container Usage at Netflix

#60

Post by mdekkers »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

Don't they also have some kind of recommendation engine? I bet that takes a bit more data and processing (who watched what, did they finish, maybe even more like what kind of scenes did they stop watching, or watched again, etc)
Post reply on HN