Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping Netflix Reliable Using Prioritized Load Shedding

netflixtechblog.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#51

Post by gameswithgo »

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

there has been the good kind of capitalism going on between the video streaming services before. earlier on I remember netflix was way better than amazon, but amazon upped their game since.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#53

Post by marta_morena_28 »

ComputerGuru wrote: post #25
bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

TCP doesn't take care of packet loss. What TCP does is make sure your packets are not lost, even if you have 99% packet loss. On the flip-side, that means that if TCP can't deliver a single packet (say out of a billion), the whole stream stops at this one packet...

Which is why TCP is a horrible choice for any streaming service and a horrible choice for lossy connections, and I would be quite surprised if Netflix relied on it. UDP is the perfect choice for streaming, since video decoders can handle packet loss pretty well. The rest you can achieve with good tradeoff between Reed-Solomon codes and key framing.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#54

Post by alaties »

marta_morena_28 wrote: post #53
ComputerGuru wrote: post #25

Earlier quoted context omitted.

This sounds like an MTU issue. TCP takes care of mere (eg probabilistic) packet loss ok. MTU issues have actually crept back up because TLS exacerbates any underlying MTU problems. IPv6 doubly so (when any hops - especially yours - don’t follow path MTU detection requirements).

TCP doesn't take care of packet loss. What TCP does is make sure your packets are not lost, even if you have 99% packet loss. On the flip-side, that means that if TCP can't deliver a single packet (say out of a billion), the whole stream stops at this one packet... Which is why TCP is a horrible choice for any streaming service and a horrible choice for lossy connections, and I would be quite surprised if Netflix rel…

I'm not sure this statement is generally true for Netflix's use case.

UDP provides no out of order packet handling which _needs_ to be handled for video streaming. UDP is by default unbuffered throughout transport and tends to cause greater stress to client systems since they need to respond per packet rather than per traffic stream (IP+port combo). As a client developer, you end up reimplementing 90-95% of what TCP gives you out of the box at great development and QA cost. You also drain battery on mobile devices with all the interrupts your causing doing UDP. The upside with a UDP-based implementation is the latency from server to client display is usually much less (tens of milliseconds vs hundreds to thousands), but the trade-offs involved are almost never worth it for a static media streaming site like Netflix.

Even dynamic media streaming sites like Twitch rarely dip into UDP server-client implementations unless there are some unusual requirements.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#55

Post by asdff »

stormbeard wrote: post #33
RyanShook wrote: post #31

How much better would the world be if the Netflix engineering team tackled real-world problems instead of making sure we can all binge watch Stranger Things? Such a smart group of people.

Specifically, what would you suggest applying this skill-set to?

A local start, given Netflix has real estate in LA: using these skills to develop better consumption services for unreliable internet users toward developing better tools for distance learning for LAUSD. Lots of students are falling behind thanks to issues with poor unreliable connectivity due to the expenses associated with having broad band internet. In march, 17% of LAUSD families had no internet at home. Today, more have bought internet plans, but given the job lossess suffered by the working poor, these are probably the cheapest internet packages available. Given how JS heavy these online education sites are, this is a poor experience for students who are already dealing with the stresses associated with being on the bottom of the economic totem pole in a high cost of living city.

https://www.latimes.com/california/story/2020-09-21/online-l...

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#56

Post by umbs »

bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

How did you measure 4 to 6% packet loss? Do you have scripts to ping some server and you are collecting packet loss data? I would like to collect such data for my home network and am curious.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#58

Post by nhumrich »

RyanShook wrote: post #31

How much better would the world be if the Netflix engineering team tackled real-world problems instead of making sure we can all binge watch Stranger Things? Such a smart group of people.

Most companies have smart engineers. The difference is how they prioritize tasks. Most companies spend a significant amount of time building new features. Netflix is pretty low in terms of number of features. Their core product is uptime and reliability. So the result is that they _have_ to reinvent scalable solutions because the rest of us are accepting production issues for more time to build out features. What you see is the result of laser focus on uptime. Not even AWS cares about uptime as much as netflix.

Re: Keeping Netflix Reliable Using Prioritized Load Shedding

#60

Post by lostmsu »

umbs wrote: post #56
bklyn11201 wrote: post #2

Totally anecdotal evidence, but I was in a rural NY house served by DSL for the past 6 months. The DSL has consistent packet loss between 4 and 6%. The only video service that could handle this level of packet loss well was Amazon Prime. Netflix couldn't even load its browse screen until the past two weeks, where something changed, and suddenly Netflix could handle the high packet loss as well as Amazon Prime. Thank…

How did you measure 4 to 6% packet loss? Do you have scripts to ping some server and you are collecting packet loss data? I would like to collect such data for my home network and am curious.

Simple ping command actually prints statistics
Post reply on HN