Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Container Usage at Netflix

techblog.netflix.com

Post reply on HN
Print view View original

151–160 of 160 posts

Re: The Evolution of Container Usage at Netflix

#151

Post by toast0 »

dominotw wrote: post #23
yeukhon wrote: post #15

Netflix Engineering team amazes me a lot. They literally took all the available apis, and build their own platform despite some of the features are already in the AWS offerings. I suppose they did it mainly because the native service isn't flexible and robust enough for their use cases. BTW, their opening positions are always prefixed with "senior" title but I guess that makes sense; Netflix builds pretty much everyt…

I asked that in an interview with Netflix, why they were building their own vs open source. The answer I got was "It doesn't work at the scale that Netflix operates" , not quite sure what that means but I didn't press him further.

Most (but not all) open source projects aren't built or tested with huge scale in mind, and often you have to do things different enough that it's not worth the effort to change an existing project when you could just build it yourself. Especially if you have a lot of custom environment thing to integrate with. A simple example is projects using MySQL rarely have support for separate read and write servers, but that's a pretty common way to scale out, and plumbing that through is going to be more painful than to do it right in the first place.

The other thing is 'early optimization' when you know millions of users are going to use something, you have to build things right to begin with.

Re: The Evolution of Container Usage at Netflix

#152

Post by user5994461 »

orf wrote: post #143
user5994461 wrote: post #138

Earlier quoted context omitted.

Forget about the joke for a minute and let's be realistic. It is entirely possible and reasonable to prototype a streaming service serving 20% of the us traffic with a single machine and mongodb. Well, a single machine, mongodb AND akamai.

Well of course, mongodb is web scale don't ya know

MongoDB is shit. No need to use it.

What's important is really the distribution and caching layer, and Akamai has been offering that for a very long time for all the big internet companies.

Re: The Evolution of Container Usage at Netflix

#153

Post by sofaofthedamned »

Diederich wrote: post #130
sofaofthedamned wrote: post #112

Earlier quoted context omitted.

Funny you should mention Walmart. I went to visit them in ~1992 when they were the biggest user of Teradata at the time. Never used Teradata after that, but i'm interviewing with them in the UK next week...

Yup. WalMart was doing big data analytics, mixed in with a lot of real-time processing, many years before those became a thing in the rest of the industry. If you don't mind me asking, what kind of position are you looking at in the UK next week?

Devops.

Yes Teradata were awesome at the time. Their secret sauce was the hashed index stuff,which I think was the source of their major patents.

Our PM was not amused when I pointed out that, although hashed indexes were awesome, they did nothing when you were doing a wildcard search at which point you're doing a table space scan. Nobody has thought about this wrinkle...

Re: The Evolution of Container Usage at Netflix

#154

Post by [deleted] »

richardwhiuk wrote: post #6
nailer wrote: post #3

They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing…

Or use something like GKE where the containers should be running directly on hardware, so you only have one layer.

[deleted]

Re: The Evolution of Container Usage at Netflix

#155

Post by manigandham »

RcouF1uZ4gsC wrote: post #52

Netflix seems so over engineered to me. They basically have a catalog of a few thousand movies that are negotiated months in advance of actual use. Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box. In regard to the analytics, t…

> Basically, they just need to encode them and put them on a box and ship them to edge caches. Caching immutable data scales incredibly well.I would also bet that 99.99% of the movies people actually watch on Netflix, would fit on a single box.

This is exactly what they do. The video serving part isn't that complicated, it's the rest of the site and APIs.

Although those are a tad over engineered considering the final output is often a slow and unwieldy UX.

Re: The Evolution of Container Usage at Netflix

#156

Post by LocalH »

emptythought wrote: post #91
zaidos wrote: post #88

Earlier quoted context omitted.

That is the core of their product. However, there are a lot of by-products and back office software needed to run the company. As a consumer, you most likely are only perceiving a small percentage of the technology running a company. - Applications for all devices Netflix is on (TVs, Phones, Displays, Roku, Apple TV, etc.) and all the support around it. - Recommendation and analytical software within Netflix. Not eve…

Another thing worth noting is that they still support basically every piece of hardware they ever did. Think built in apps on old smart TVs or early blu-ray decks that haven't been patched in YEARS, devices from before 2009~ even. You can still watch netflix on an original Wii. That hasn't gotten any OS or software patches since like 2011. That's a long tail of legacy clients to have to deal with.

For the record, the official Netflix client for PS2 (only legitimately released in some South American countries) was discontinued and is unsupported. I'm guessing they moved exclusively to H.264 at some point (no idea what codec they served prior to that point, but to be playable on the PS2 it would have either had to be MPEG-2 or MPEG-4 ASP/Xvid)

Re: The Evolution of Container Usage at Netflix

#157

Post by scurvy »

gtirloni wrote: post #81
scurvy wrote: post #44

Earlier quoted context omitted.

Doesn't kubernetes run an overlay network (at least in some scenarios)?

An overlay network is not a requirement. The only requirement is that pods (collection of containers) should be able to communicate with each other directly without NAT. Each pod gets an IP address in the container network. Overlay network technologies (flannel, weave, calico, etc) are popular but they aren't mandatory. You can implement it using hardware switches and VLANs if you wish.

Sorry, but what are you inferring by hardware switches? Something not an overlay? Something that switches VXLAN in hardware?

Re: The Evolution of Container Usage at Netflix

#158

Post by Diederich »

sofaofthedamned wrote: post #153
Diederich wrote: post #130

Earlier quoted context omitted.

Yup. WalMart was doing big data analytics, mixed in with a lot of real-time processing, many years before those became a thing in the rest of the industry. If you don't mind me asking, what kind of position are you looking at in the UK next week?

Devops. Yes Teradata were awesome at the time. Their secret sauce was the hashed index stuff,which I think was the source of their major patents. Our PM was not amused when I pointed out that, although hashed indexes were awesome, they did nothing when you were doing a wildcard search at which point you're doing a table space scan. Nobody has thought about this wrinkle...

Devops with WalMart in UK...so with ASDA then?

Re: Teradata: I sat next to that group for half a year, but I had no other direct exposure to it. I did what one might call 'devops' for Network Engineering.

Re: The Evolution of Container Usage at Netflix

#159

Post by sofaofthedamned »

Diederich wrote: post #158
sofaofthedamned wrote: post #153

Earlier quoted context omitted.

Devops. Yes Teradata were awesome at the time. Their secret sauce was the hashed index stuff,which I think was the source of their major patents. Our PM was not amused when I pointed out that, although hashed indexes were awesome, they did nothing when you were doing a wildcard search at which point you're doing a table space scan. Nobody has thought about this wrinkle...

Devops with WalMart in UK...so with ASDA then? Re: Teradata: I sat next to that group for half a year, but I had no other direct exposure to it. I did what one might call 'devops' for Network Engineering.

No, this was for WH Smith. We went to Walmart to see how they used it. It was awesome, but misunderstood.

Re: The Evolution of Container Usage at Netflix

#160

Post by lima »

nailer wrote: post #53
lima wrote: post #45

Earlier quoted context omitted.

SELinux is NOT isolation. The main issue is the large kernel attack surface and SELinux, while important, only solves a small part of that.

That's true, but a lot of the classic containers-not-containing issues (/sysfs hacks to get into the parent kernel etc) are prevented by SELinux policies. See https://blog.openshift.com/securing-dockers-future-with-seli...

Agreed.

But people keep selling SELinux or AppArmor as a solution for multi-tenant container environments, which is just plain false.

The real solution are efforts like like Intel's Clear Containers and Hyper's runV.

Post reply on HN