Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Container Usage at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

111–120 of 160 posts

Re: The Evolution of Container Usage at Netflix

#111

Post by RcouF1uZ4gsC »

zaidos wrote: post #88
RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

That is the core of their product. However, there are a lot of by-products and back office software needed to run the company. As a consumer, you most likely are only perceiving a small percentage of the technology running a company. - Applications for all devices Netflix is on (TVs, Phones, Displays, Roku, Apple TV, etc.) and all the support around it. - Recommendation and analytical software within Netflix. Not eve…

Most of these stuff are also done by large corporate IT departments such as banks. They all consider it pretty routine and are not writing engineering blog posts.

Re: The Evolution of Container Usage at Netflix

#112

Post by sofaofthedamned »

Diederich wrote: post #39
dominotw wrote: post #23

Earlier quoted context omitted.

I asked that in an interview with Netflix, why they were building their own vs open source. The answer I got was "It doesn't work at the scale that Netflix operates" , not quite sure what that means but I didn't press him further.

It's rather rare to see off the shelf technologies (open or closed) that scale to the top 5%. This makes sense because the techniques required at that level are pretty different, and don't apply well to other situations. So, making an off the shelf product that actually does scale to the top 5% rarely makes sense, since, at best, the customer base will be limited. In the average case, though, the customer base will b…

Funny you should mention Walmart. I went to visit them in ~1992 when they were the biggest user of Teradata at the time. Never used Teradata after that, but i'm interviewing with them in the UK next week...

Re: The Evolution of Container Usage at Netflix

#113

Post by acdha »

pm90 wrote: post #107
acdha wrote: post #96

Earlier quoted context omitted.

Even if that claim wasn't wrong, it's an unrelated question. If there were rate-limiting problems, they'd apply to using EC2 at all even without involving containers.

I think you're ignoring the fundamental issue when deploying container based services v/s services on a multiple VM's. Usually, the architecture for containers involves spinning up a bunch of VM's and deploying some kind of layer on top of that (either K8s or Swarm or something else). When you deploy containers, they may not be on the same VM, or the overlay network itself may require some kind of communication to an…

Do you have any evidence of this rate-limiting showing that it's that much of a problem? People have been running clustered apps on EC2 for over a decade and it's not like you hear people saying you can't run Cassandra, ElasticSearch, etc. on EC2 because the network is limited.

Similarly, do you have any data showing that a container system has such incredible overhead compared to the actual application workload? I mean, if that was true you'd think the entire Kubernetes team would be staying up nights figuring out how to reduce overhead.

Re: The Evolution of Container Usage at Netflix

#114

Post by RightMillennial »

geodel wrote: post #13
hueving wrote: post #2

>The theme that underlies all these improvements is developer innovation velocity I can't wait until this becomes buzzword de jour and startups start using it in their product descriptions. Then someone needs to start talking about products to "enable developer innovation acceleration" to outpace these crufty companies stuck at 25 kph.

Yup. I think further down someone will 'democratize innovation velocity and acceleration'

But watch out for the innovative jerks. They may mess up the acceleration.

Re: The Evolution of Container Usage at Netflix

#115

Post by savanaly »

rsync wrote: post #98
lmkg wrote: post #62

Earlier quoted context omitted.

At one point, Netflix video streaming accounted for something absurd like 20% of all Internet traffic in the country. Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling. Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing…

"Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing to shuttle 5GB of data with no hiccups for an hour." Speaking of which ... I notice that while youtube continues to buffer video while paused, netflix (and many, many other video players online) do not. So while netflix will auto-adjust…

Probably too many cases where they deliver a bunch of content while it's paused and then the content never gets played. More than half the people who pause it probably end up closing the tab without finishing the video, so one can save a lot of bandwidth by only delivering the data when it's needed. A totally unsupported hypothesis.

Re: The Evolution of Container Usage at Netflix

#116

Post by lowbloodsugar »

dominotw wrote: post #31

> We implemented multi-tenant isolation (CPU, memory, disk, networking and security) using a combination of Linux, Docker and our own isolation technology. Curious what their 'own isolation technology' does that docker doesn't. Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this i…

Fenzo is a java library for deciding how to allocate tasks when offered resources. It makes implementing Mesos frameworks easier because it turns out the question: "Given an offer of X resources, and a list of tasks that need running, what is the best use of the offered resources?" is actually quite hard. (3D knapsack hard).

"Apache Mesos frameworks match and assign resources to pending tasks. Fenzo presents a plugin-based, Java library that facilitates scheduling resources to tasks by using a variety of possible scheduling objectives, such as bin packing, balancing across resource abstractions (such as AWS availability zones or data center racks), resource affinity, and task locality." [1]

Marathon is a Mesos framework for scheduling long running applications (like rest services) and keeping them running. If what you want to do is serve http traffic then Marathon does the job (although the stand-alone UI is now deprecated and will only bug-fixed for "the next few months", so you'd better like the full DC/OS offering).

Titus appears to combine the functionality of Marathon, plus the ability to run batch jobs. I wondered if Titus was a fork of Marathon with new bits, but that doesn't appear to be the case. I believe it deals with one glaring flaw in Mesos, which is that frameworks all independently calculate the best use of the offered resources. When compute becomes available, Mesos makes offers to frameworks, with some basic logic such as making offers to frameworks currently consuming the least. But that means that there is no way to customize the resource allocation across different use cases (e.g. between REST APIs, one-off tasks and Spark clusters). It'd be great if Fenzo did "sit on top of Marathon", so I could customize how it schedules based on the "bigger picture". Titus avoids the problem because it schedules everything.

[1] https://github.com/Netflix/Fenzo

Re: The Evolution of Container Usage at Netflix

#117

Post by [deleted] »

hueving wrote: post #2

>The theme that underlies all these improvements is developer innovation velocity I can't wait until this becomes buzzword de jour and startups start using it in their product descriptions. Then someone needs to start talking about products to "enable developer innovation acceleration" to outpace these crufty companies stuck at 25 kph.

[deleted]

Re: The Evolution of Container Usage at Netflix

#118

Post by johnsmith21006 »

throwasehasdwi wrote: post #47
jo909 wrote: post #41

Earlier quoted context omitted.

Could you elaborate on why containers don't perform on EC2? I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is networ…

Containers are simply processes so NOT 1% overhead! No overhead but get hard to believe.

Re: The Evolution of Container Usage at Netflix

#119

Post by johnsmith21006 »

What has happened to HN? This thread is so filled with misinformation it is clear there is very little understanding of container.

A container is just a process. Really no different than any other. So cache and shared libraries, etc all the same with just a little care.

Re: The Evolution of Container Usage at Netflix

#120

Post by mentat »

pm90 wrote: post #107
acdha wrote: post #96

Earlier quoted context omitted.

Even if that claim wasn't wrong, it's an unrelated question. If there were rate-limiting problems, they'd apply to using EC2 at all even without involving containers.

I think you're ignoring the fundamental issue when deploying container based services v/s services on a multiple VM's. Usually, the architecture for containers involves spinning up a bunch of VM's and deploying some kind of layer on top of that (either K8s or Swarm or something else). When you deploy containers, they may not be on the same VM, or the overlay network itself may require some kind of communication to an…

You run a compute pool, you don't spin up EC2 instances on demand for this kind of application. You scale the pool based on target utilization metrics.
Post reply on HN