Keeping Netflix Reliable Using Prioritized Load Shedding • Hacker News Board

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#41

Post by stingraycharles » Mon, Nov 02, 2020, 8:03 PM UTC

ComputerGuru wrote: post #25

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

Aren’t MTU issues typically only up to a router? As in, even if the parent had a different MTU than Netflix uses, it wouldn’t matter since their router or the ISP’s router will transform packets between their appropriate MTUs?

And if this is true, then how could it be that Amazon works without problem and Netflix doesn’t?

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#42

Post by judge2020 » Mon, Nov 02, 2020, 8:15 PM UTC

closeparen wrote: post #39

madeofpalk wrote: post #17

Earlier quoted context omitted.

Seperate annecdote - I worked on an inflight satelite wifi project and I was surprised at how well both Youtube and Netlifx worked over a medium-bandwidth/high-latency connection. Granted, we had specific QoS/traffic shaping to improve reliability without gobbling up all the bandwidth (stream Netflix was an advertised feature of the wifi service), but it still seemed like magic.

Wow! You actually stream Netflix to an airplane? I always guessed that inflight VOD services had the movies stored in a server on the plane.

Things like the Delta “gogo in-flight entertainment” do store their movies on the plane, but people will want to watch their Netflix/Prime/etc content on the plane as well.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#43

Post by legulere » Mon, Nov 02, 2020, 8:18 PM UTC

ComputerGuru wrote: post #25

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

Isn’t streaming done usually via UDP?

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#44

Post by stingraycharles » Mon, Nov 02, 2020, 8:35 PM UTC

legulere wrote: post #43

ComputerGuru wrote: post #25

Earlier quoted context omitted.

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

Isn’t streaming done usually via UDP?

It’s typically all HTTP requests; nowadays with HTTP3 we are back to using UDP, but apart from real-time video conferencing etc I don’t believe many streaming services use anything other than HTTP.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#45

Post by erulabs » Mon, Nov 02, 2020, 9:00 PM UTC

RyanShook wrote: post #31

How much better would the world be if the Netflix engineering team tackled real-world problems instead of making sure we can all binge watch Stranger Things? Such a smart group of people.

How do you know what a real-world problem is or isn’t? Avoid the intellectual trap of trying to socially engineer the world to your tastes. Watching stranger things might be more important than you realize.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#46

Post by madeofpalk » Mon, Nov 02, 2020, 9:10 PM UTC

closeparen wrote: post #39

madeofpalk wrote: post #17

Earlier quoted context omitted.

Seperate annecdote - I worked on an inflight satelite wifi project and I was surprised at how well both Youtube and Netlifx worked over a medium-bandwidth/high-latency connection. Granted, we had specific QoS/traffic shaping to improve reliability without gobbling up all the bandwidth (stream Netflix was an advertised feature of the wifi service), but it still seemed like magic.

Wow! You actually stream Netflix to an airplane? I always guessed that inflight VOD services had the movies stored in a server on the plane.

Both.

There’s “inflight entertainment” where all the movies/shows are indeed stored locally on the plane, with either seatback or custom/white label streaming app for BYOD.

But in addition they were advertising streaming Netflix and YouTube over the satellite WiFi.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#47

Post by tyingq » Mon, Nov 02, 2020, 9:14 PM UTC

stingraycharles wrote: post #41

ComputerGuru wrote: post #25

Earlier quoted context omitted.

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

Aren’t MTU issues typically only up to a router? As in, even if the parent had a different MTU than Netflix uses, it wouldn’t matter since their router or the ISP’s router will transform packets between their appropriate MTUs? And if this is true, then how could it be that Amazon works without problem and Netflix doesn’t?

"how could it be that Amazon works without problem and Netflix doesn’t"

Supporting Path MTU discovery (PMTUD), or perhaps just capping their outbound packets to 1450 or similar. Cloudflare found and fixed a problem in this space: https://blog.cloudflare.com/path-mtu-discovery-in-practice/

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#48

Post by TheSwordsman » Mon, Nov 02, 2020, 9:34 PM UTC

khalilravanna wrote: post #15

annoyingnoob wrote: post #10

According to the diagram, Netflix is injecting chaos into the chaos control panel. Is that right? Looks like the arrow goes the wrong direction.

I’m wondering if it’s more of a “Chaos Injector” component/service that reads configuration data from the Chaos Control Plane on what to target, with parameters on how/when to do so. That would make the arrow make sense in my mind given it sounds like that’s a solid pattern for scaling these data/control plane flows: https://aws.amazon.com/builders-library/avoiding-overload-in...

This. It's an internal system called ChAP, Chaos Automation Platform. It has the ability to target failure down to specific RPC calls in single instances, using platform components that services consume as the mechanism for doing that injection.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#49

Post by xxpor » Mon, Nov 02, 2020, 9:35 PM UTC

ComputerGuru wrote: post #25

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

>TCP takes care of mere (eg probabilistic) packet loss ok.

I'd imagine this is largely due to MSS clamping rather than actual MTU caused packet loss.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#50

Post by tmpz22 » Mon, Nov 02, 2020, 9:45 PM UTC

ComputerGuru wrote: post #27

tmpz22 wrote: post #13

A lot of websites will now fail requests early based on a timeout, forcing users to refresh the page. I have to wonder if ad-based sites enjoy this behavior because it could lead to more ad impressions. Talking about you reddit.

I think you’re talking about SPAs in specific. Many have race conditions in frontend code that are not revealed on fast connections or when all resources are loaded with the same speed/consistency. Open the developer console next time it happens, I bet you’ll find a “foo is not a function” or similar error caused by something not having init yet and the code not properly awaiting it. If an SPA core loop errors out, l…

You see it in backend code too. For example Golang's context.WithTimeout is used to time out http requests and database calls that may be taking too long. This is particularly irksome with microservices where multiple services are running timeouts that interfere with one another.

It is becoming du jour to quell 99 percentile latency spikes (i.e. 1:100 requests will take substantially longer) by terminating the requests, which may not always be in the best interest of the user even if it is convenient for the devops teams and their promotion packets.