The Evolution of Container Usage at Netflix • Hacker News Board

Re: The Evolution of Container Usage at Netflix

#131

Post by sheeshkebab » Fri, Apr 21, 2017, 8:59 PM UTC

johansch wrote: post #125

yomly wrote: post #89

Earlier quoted context omitted.

Beat me to it. I am sure a large part Netflix's success is how seamless its products are: there's minimal to no loading time and the video stream is virtually always smooth - including during times of action/high frame rate. Creating a video streaming platform is one thing, but creating a content delivery platform that is effectively as reliable as TV that is also cost efficient is no small task if you ask me...

Bullcrap. The tech is good enough (but not that much better beyond that level). >90% of the success comes from the content creation/licensing strategy/execution and marketing. I.e. non-tech. At this point it's a bit like arguing that Disney's great success in Bluray sales comes from their encoding/Bluray mastering expertise.

Tech part was "good enough" for last 20+ years - it's why most companies call their tech teams "IT department".

There is however something about that last 5-10% where moving a few bits around the right way gives you business advantage, over places like Disney.

That said, it does seem like some of this Netflix stuff is overbuilt, although I don't need to run tens of 1000's of ec2, so maybe it's worth it.

Re: The Evolution of Container Usage at Netflix

#132

Post by sethev » Fri, Apr 21, 2017, 9:10 PM UTC

johansch wrote: post #125

yomly wrote: post #89

Earlier quoted context omitted.

Beat me to it. I am sure a large part Netflix's success is how seamless its products are: there's minimal to no loading time and the video stream is virtually always smooth - including during times of action/high frame rate. Creating a video streaming platform is one thing, but creating a content delivery platform that is effectively as reliable as TV that is also cost efficient is no small task if you ask me...

Bullcrap. The tech is good enough (but not that much better beyond that level). >90% of the success comes from the content creation/licensing strategy/execution and marketing. I.e. non-tech. At this point it's a bit like arguing that Disney's great success in Bluray sales comes from their encoding/Bluray mastering expertise.

You seem to stumbling into the distinction between functional and non-functional requirements. Netflix spends massively more on content then tech so that's consistent with what you're saying. To use you're Disney example though, yes it would effect their success if they weren't able to deliver their content because their Bluray mastering expertise was lacking.

Re: The Evolution of Container Usage at Netflix

#133

Post by rhizome » Fri, Apr 21, 2017, 9:22 PM UTC

orf wrote: post #109

always_good wrote: post #103

Earlier quoted context omitted.

"Netflix is over-engineered, it's just videos lol" Classic HN hubris. For example, Netflix dynamically adjusts video quality on demand. Pretty easy feature. The top HN comment will explain how they would implement that in five minutes -- hypothetically, of course!

"Responsible 20% of the entire us internet traffic? Solved problem mate, i could do it on a single machine with mongodb. Overengineered!"

Just think how much more they could do if they stopped using MySQL. /s

Re: The Evolution of Container Usage at Netflix

#134

Post by rhizome » Fri, Apr 21, 2017, 9:25 PM UTC

therealdrag0 wrote: post #128

aquilaFiera wrote: post #69

Earlier quoted context omitted.

This sounds like one of those developer estimates that takes 10x longer than stated.

Anyone up for the "make Netflix in a weekend challenge"?

Rails + Devise + Paperclip + JWPlayer = who's got Bob Iger's phone number?

Re: The Evolution of Container Usage at Netflix

#135

Post by jperras » Fri, Apr 21, 2017, 9:44 PM UTC

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

> A national grocery chain, probably has a similar amount of data ingestion and a similar analytics need.

It's off-topic, but having interacted with national-level grocery chains before, I think you grossly over-estimate their technological capacity and savvy. They're probably one of the slowest industries to adopt new technological advancements, for better or worse.

Re: The Evolution of Container Usage at Netflix

#136

Post by user5994461 » Fri, Apr 21, 2017, 9:59 PM UTC

lmkg wrote: post #62

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

At one point, Netflix video streaming accounted for something absurd like 20% of all Internet traffic in the country. Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling. Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing…

> Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling.

I would imagine there is not. When you master something for 1%, you do 20% without too much hassle. (Let's remember that netflix is only serving a couple of videos to a lot of customers, that is embarrassingly parallel and cachable).

About reliability. Video is one of the thing that has the least need for reliability. You can have thousands of hiccup during the hours, doesn't matter, nooone can see any of them.

I expect most of the tech challenges to be in cost optimizations, not in delivering the service.

Re: The Evolution of Container Usage at Netflix

#137

Post by user5994461 » Fri, Apr 21, 2017, 10:04 PM UTC

sheeshkebab wrote: post #131

johansch wrote: post #125

Earlier quoted context omitted.

Bullcrap. The tech is good enough (but not that much better beyond that level). >90% of the success comes from the content creation/licensing strategy/execution and marketing. I.e. non-tech. At this point it's a bit like arguing that Disney's great success in Bluray sales comes from their encoding/Bluray mastering expertise.

Tech part was "good enough" for last 20+ years - it's why most companies call their tech teams "IT department". There is however something about that last 5-10% where moving a few bits around the right way gives you business advantage, over places like Disney. That said, it does seem like some of this Netflix stuff is overbuilt, although I don't need to run tens of 1000's of ec2, so maybe it's worth it.

Tens of thousands of ec2 instances, none of which served any customer, if you red the article carefully.

Re: The Evolution of Container Usage at Netflix

#138

Post by user5994461 » Fri, Apr 21, 2017, 10:09 PM UTC

orf wrote: post #109

always_good wrote: post #103

Earlier quoted context omitted.

"Netflix is over-engineered, it's just videos lol" Classic HN hubris. For example, Netflix dynamically adjusts video quality on demand. Pretty easy feature. The top HN comment will explain how they would implement that in five minutes -- hypothetically, of course!

"Responsible 20% of the entire us internet traffic? Solved problem mate, i could do it on a single machine with mongodb. Overengineered!"

Forget about the joke for a minute and let's be realistic.

It is entirely possible and reasonable to prototype a streaming service serving 20% of the us traffic with a single machine and mongodb.

Well, a single machine, mongodb AND akamai.

Re: The Evolution of Container Usage at Netflix

#139

Post by user5994461 » Fri, Apr 21, 2017, 10:22 PM UTC

nailer wrote: post #8

bertlequant wrote: post #4

I would've imagined Netflix had their own hardware, both compute and storage.

I've heard they're looking at 'architecture in a box' solutions, which would allow them to abstract designs from specific cloud providers and perform cloud arbitrage; eg, if Cloud Provider X can run the app at 14c/hour, and Cloud Provider Y can run the app at 22c/hour, then they can just deploy the entire architecture on Cloud Provider Y.

Another simple problem that's solved: Auto scaling of stateless services across thousands of machines, in multiple datacenter and cloud providers all around the world.

https://www.nomadproject.io/

There are also kubernetes and mesosphere to do similar thing, but they are harder to use, you can't learn them over a week end.

Re: The Evolution of Container Usage at Netflix

#140

Post by lkrubner » Fri, Apr 21, 2017, 10:32 PM UTC

This is some amazing scale:

-------------------

We run a peak of 500 r3.8xl instances in support of our batch users. That represents 16,000 cores of compute with 120 TB of memory. We also added support for GPUs as a resource type using p2.8xl instances to power deep learning with neural nets and mini-batch.

In the early part of 2017, our stream-processing-as-a-service team decided to leverage Titus to enable simpler and faster cluster management for their Flink based system. This usage has resulted in over 10,000 service job containers that are long running and re-deployed as stream processing jobs are changed. These and other services use thousands of m4.4xl instances.

While the above use cases are critical to our business, issues with these containers do not impact Netflix customers immediately. That has changed as Titus containers recently started running services that satisfy Netflix customer requests.