Keeping Netflix Reliable Using Prioritized Load Shedding • Hacker News Board

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#21

Post by baq » Mon, Nov 02, 2020, 5:56 PM UTC

timhaak wrote: post #16

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

Slightly unrelated but does the packet loss happen all the time or when close to maximum of the line. Used to have similar problems with an ADSL line but found if I limited the line (Both up and down) I could find a magic number where the packet loss went away. (Well most of the time :)) Though it did need to be tuned for different times of they . ie high congestion times need it to be lower. Though technically it sh…

This is normal if your router doesn’t prioritize control traffic. A rate limit allows all the ACKs to normally leave your network instead of getting queued up.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#22

Post by kevincox » Mon, Nov 02, 2020, 6:06 PM UTC

crazygringo wrote: post #20

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

> Netflix couldn't even load its browse screen until the past two weeks I assume the browse screen is based entirely on TCP? I'm struggling to understand why packet loss would prevent it from loading -- it should be slower but TCP should handle re-transmission, no? Or is Netflix doing something tricky with UDP even in their browsing UX?

If I had to guess they probably had timeouts that were too aggressive. Client timeouts are a very hard problem because it is difficult to tell the difference between "working, but slowly" and "something went wrong, the best bet is to try again".

Back in the day we used to have timeouts based on individual reads/writes which will often better answer "is this HTTP request making progress". However the problem with these sort of timeouts is they don't compose well so most people end up having an end-to-end deadline.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#23

Post by [deleted] » Mon, Nov 02, 2020, 6:10 PM UTC

crazygringo wrote: post #20

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

> Netflix couldn't even load its browse screen until the past two weeks I assume the browse screen is based entirely on TCP? I'm struggling to understand why packet loss would prevent it from loading -- it should be slower but TCP should handle re-transmission, no? Or is Netflix doing something tricky with UDP even in their browsing UX?

[deleted]

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#24

Post by seized » Mon, Nov 02, 2020, 6:17 PM UTC

madeofpalk wrote: post #17

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

Seperate annecdote - I worked on an inflight satelite wifi project and I was surprised at how well both Youtube and Netlifx worked over a medium-bandwidth/high-latency connection. Granted, we had specific QoS/traffic shaping to improve reliability without gobbling up all the bandwidth (stream Netflix was an advertised feature of the wifi service), but it still seemed like magic.

When Plex rolled out it's auto quality/auto bandwidth adjustment it actually worked very well over airplane satellite wifi as well. I watched a few things from my own server.

I'm amazed that service allowed streaming though...

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#25

Post by ComputerGuru » Mon, Nov 02, 2020, 6:23 PM UTC

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#26

Post by bklyn11201 » Mon, Nov 02, 2020, 6:27 PM UTC

timhaak wrote: post #16

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

Slightly unrelated but does the packet loss happen all the time or when close to maximum of the line. Used to have similar problems with an ADSL line but found if I limited the line (Both up and down) I could find a magic number where the packet loss went away. (Well most of the time :)) Though it did need to be tuned for different times of they . ie high congestion times need it to be lower. Though technically it sh…

It happens nearly all the time. We use very little DSL bandwidth but are quite rural (miles from primary telephone infrastructure)

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#27

Post by ComputerGuru » Mon, Nov 02, 2020, 6:27 PM UTC

tmpz22 wrote: post #13

A lot of websites will now fail requests early based on a timeout, forcing users to refresh the page. I have to wonder if ad-based sites enjoy this behavior because it could lead to more ad impressions. Talking about you reddit.

I think you’re talking about SPAs in specific. Many have race conditions in frontend code that are not revealed on fast connections or when all resources are loaded with the same speed/consistency. Open the developer console next time it happens, I bet you’ll find a “foo is not a function” or similar error caused by something not having init yet and the code not properly awaiting it. If an SPA core loop errors out, load will be halted or even a previously loaded or partially loaded page will become blank or partially so. Refreshing it will load already retrieved resources from cache and often “fixes” the problem.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#28

Post by ComputerGuru » Mon, Nov 02, 2020, 6:28 PM UTC

abalone wrote: post #11

For me this link just opens the Medium app and fails to load the article. I had to force it to open in a browser. Seems like a pretty bad Medium bug.

Obvious suggestion but not made in snark: uninstall the medium app? I’ve had to do that for lots of poorly developed apps or apps developed not in sync with the web frontend.

Edit: it is a bad link and I can see why this would happen if you had the Medium app installed. It’s a “branded” Medium post (i.e. appears on the Netflix-owned domain) but clicking the link redirects you to medium.com then redirects you back to the cname.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#29

Post by jeffbee » Mon, Nov 02, 2020, 6:29 PM UTC

kache_ wrote: post #14

When deciding what mechanism to employ to load shed, you should keep in mind the layer at which you are load shedding. Modern distributed systems are comprised of many layers. You can do it at the load balancer, at the OS level, or in the application logic. This becomes a trade-off. As you get closer to the core application logic, the more information you will have to make a decision. On the other hand, as you get cl…

You touch on the key thing that people sometimes overlook. Whatever you are doing to serve errors has to be strictly less expensive than serving successes. If your load shedding error path does things like logging synchronously to a file (as you might get from a logging library that synchronizes outputs for warnings and errors, but not information), taking a lock to update a global error counter, or formatting stack traces in exceptions, it's possible that load shedding will _cause_ the collapse of your service instead of preventing it.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#30

Post by fkskdkgif » Mon, Nov 02, 2020, 7:01 PM UTC

timhaak wrote: post #16

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

Slightly unrelated but does the packet loss happen all the time or when close to maximum of the line. Used to have similar problems with an ADSL line but found if I limited the line (Both up and down) I could find a magic number where the packet loss went away. (Well most of the time :)) Though it did need to be tuned for different times of they . ie high congestion times need it to be lower. Though technically it sh…

Dropped packets are often a symptom that the MTU value is set too high. That would be uncorrelated to congestion, though.