Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Container Usage at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

61–70 of 160 posts

Re: The Evolution of Container Usage at Netflix

#61

Post by pkaye »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

I can see this kind of over engineering in all the companies that hit the VC/IPO lottery. They all hire like crazy the best engineers and have to keep them busy doing challenging engineering stuff. And managers want to protect their turf so they join in. So you have ever more complex designs in anticipation of growth, side projects, detailed analysis, blogs with pretty graphs, etc. A combination of Parkinson's Law and Second System Effect.

Re: The Evolution of Container Usage at Netflix

#62

Post by lmkg »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

At one point, Netflix video streaming accounted for something absurd like 20% of all Internet traffic in the country. Regardless of whether it "scaled incredibly well," I would imagine there are still novel issues with that much scaling.

Reliability also matters differently for video than it does for normal web traffic. It's one thing to shuttle 5GB (or whatever) of data over the course of an hour. It's another thing to shuttle 5GB of data with no hiccups for an hour. Detecting and routing around machine or network issues fast enough that real-time video playback is not impeded sounds to me like a difficult challenge.

Re: The Evolution of Container Usage at Netflix

#63

Post by gbrown_ »

Is anyone else having issues accessing this?

    $ curl http://techblog.netflix.com/2017/04/the-evolution-of-container-usage-at.html
    
    
    
    Access Denied
    body {margin:0;font-family:verdana,sans-serif;} h1 {margin:0;padding:12px 25px;background-color:#343434;color:#ddd} p {margin:12px 25px;} strong {color:#E0042D;}
    
    
    Access Denied
    

    You are attempting to access a forbidden site.


    Consult your system administrator for details.

Re: The Evolution of Container Usage at Netflix

#64

Post by macNchz »

colanderman wrote: post #38
skrater wrote: post #34

Earlier quoted context omitted.

Justifying high engineering salaries, i see this a lot in great companies.

Netflix is the last place I'd expect to do this. They're famous for letting people go when they're no longer needed.

That sounds like a great incentive to build big proprietary systems that depend on your presence...

Re: The Evolution of Container Usage at Netflix

#65

Post by JustSomeNobody »

tybit wrote: post #20
yeukhon wrote: post #15

Netflix Engineering team amazes me a lot. They literally took all the available apis, and build their own platform despite some of the features are already in the AWS offerings. I suppose they did it mainly because the native service isn't flexible and robust enough for their use cases. BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everyt…

They specifically only hire seniors, they don't want to provide positions for juniors as they would prefer to pay more for more seniors than pay indirectly for training juniors. I think this reflects poorly on them, just as much as companies that use OSS but don't contribute anything back.

I guess "experience discrimination" is a thing now?

Seriously, if a company only wants to hire experienced folk, then so be it as long as they don't exercise REAL discrimination.

Re: The Evolution of Container Usage at Netflix

#66

Post by thealfreds »

gbrown_ wrote: post #63

Is anyone else having issues accessing this? $ curl http://techblog.netflix.com/2017/04/the-evolution-of-container-usage-at.html Access Denied body {margin:0;font-family:verdana,sans-serif;} h1 {margin:0;padding:12px 25px;background-color:#343434;color:#ddd} p {margin:12px 25px;} strong {color:#E0042D;} Access Denied You are attempting to access a forbidden site. Consult your system administrator for details.

Work filtering out netflix.com? Old place use to do it and it would bug me I couldn't read their techblog at work.

Re: The Evolution of Container Usage at Netflix

#67

Post by sethev »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

How does the fact that other companies have streaming video services lead to Netflix not having to solve that problem? This seems a bit like saying that Walmart doesn't need to worry about managing their supply chain because Cosco has already solved that problem.

Also if you've ever tried a TV channel's streaming service you'll realize that recording the start and end time won't tell you much about the quality of the experience.

Re: The Evolution of Container Usage at Netflix

#68

Post by tyingq »

dominotw wrote: post #31

> We implemented multi-tenant isolation (CPU, memory, disk, networking and security) using a combination of Linux, Docker and our own isolation technology. Curious what their 'own isolation technology' does that docker doesn't. Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top of marathon and sends it some sort of recommendations for scheduling. I need to find a good example of what this i…

>Curious what their 'own isolation technology' does that docker doesn't.

Probably related to that "and security" part. That isn't currently docker's strong suit.

Re: The Evolution of Container Usage at Netflix

#69

Post by aquilaFiera »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

This sounds like one of those developer estimates that takes 10x longer than stated.

Re: The Evolution of Container Usage at Netflix

#70

Post by jefe_ »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

I get the impression they're running analytics not just on the users but also the content (which would take more horsepower). Guessing they use the analysis to model scripts and generate ideas for their Original Series. Thinking of what is needed to do this, you need to analyze themes, characters, story lines, visual effects, music, many components, and then tie them to user types and viewing patterns. Then they create series with either wide appeal or strong appeal to a very small niche. On top of that, these shows load almost instantly while you're still determining whether to watch or not.
Post reply on HN