Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

171–180 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#171

Post by lanstin »

andrewf wrote: post #169
lanstin wrote: post #152

Earlier quoted context omitted.

Also, it means you are married to the reboot process. If you loose control of your memory management process too much, you'll never be able to fix it absent a complete rewrite. I worked at a place that had a lot of (c++) CGI programs with a shocking level of disregard for freeing memory, but that was ok because when the CGI request was over the process restarted. But then they reused that same code in SOA/long lived…

You kinda want some machines to reboot less frequently than others, so issues don't creep up on you. You also want some machines to reboot much more frequently than others, so you catch boot issues before they affect your entire fleet.

Not rebooting after software upgrade is an oft repeated mistake.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#172

Post by TZubiri »

bobdvb wrote: post #139
TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

Netflix has done massive amounts of work on BSD to improve it's network throughput, that's part of them enabling their file delivery from their CDN appliances. https://people.freebsd.org/~gallatin/talks/euro2022.pdf They've also contributed significantly to open source tools for video processing, one of the biggest things that stands out is probably their VMAF tool for quantifying perceptual quality in video. It's pr…

Yeah I stand corrected. Video being one of the highest entropy types of data probably means they face state of the art throughput challenges. Which are inherently tied to cost and monetization.

That said, free apps like tiktok and youtube probably face higher throughput, so the user-pays model probably means netflix is at the state of the art at high volume quality (both app experience and content) rather than sheer volume low quality or premium quality low volume markets.

I mean serving millions of customers at 8 bucks per month. Which is not quite like serving billions.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#173

Post by TZubiri »

PittleyDunkin wrote: post #155
nicce wrote: post #115

Earlier quoted context omitted.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Is it? It's pretty rare to download assets from servers that you're uploading to. Sometimes you have truly interactive app servers but that's a pretty small percentage of web traffic. Shared state is not the typical problem to solve on the internet, though it is a popular one to discuss.

That's true as the internet becomes more $ focused, companies become more interested in shoving messages (ads/propaganda) than letting users say anything). Even isp plans have asymmetric specs.

I'm honestly much more impressed by free apps like youtube and tiktok in terms of throughput, they have MUCH more traffic since users don't pay!

Re: We built a self-healing system to survive a concurrency bug at Netflix

#174

Post by TZubiri »

PittleyDunkin wrote: post #156
nicce wrote: post #140

Earlier quoted context omitted.

Yes, but it is still counts only a fraction of the purpose of their infrastructure. There are no hard global real-time sync requirements. > When you're watching a movie your progress is constantly updated, posting data This can be implemented on server side and with read requests only. A proper comparison would be YouTube where people upload videos and comment stuff in real-time.

> A proper comparison would be YouTube where people upload videos and comment stuff in real-time. Even in this one sentence you're conflating two types of interaction. Surely downloading videos is yet a third, and possibly the rest of the assets on the site a fourth. Why not just say the exact problem you think is worth of discussion with your full chest if you so clearly have one in mind?

I'd make a distinction on:

-the entropy of the data: a video is orders of magnitude than browsing metadata.

- the compute required: other than an ML algorithm optimizing for engagement, there's no computationally intensive business domain work (throughput related challenges dont count)

- finally programming complexity, in terms of business domain, is not there.

I mean my main argument is that a video provider is a simple business requirement. Sure you can make something simple at huge scale and that is a challenge. Granted.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#175

Post by TZubiri »

nicce wrote: post #140
jacksontheel wrote: post #137

Earlier quoted context omitted.

Every time you like/dislike/watchlist a movie you're posting data. When you're watching a movie your progress is constantly updated, posting data. Simple stuff but there's possibly hundreds of thousands of concurrent users doing that at any given moment.

Yes, but it is still counts only a fraction of the purpose of their infrastructure. There are no hard global real-time sync requirements. > When you're watching a movie your progress is constantly updated, posting data This can be implemented on server side and with read requests only. A proper comparison would be YouTube where people upload videos and comment stuff in real-time.

There's many cases where server side takes you 98% there, but it still makes economical sense to spend shitloads on getting that 2% there.

In this case it's not the same whether your server sends a 10 second packet and the viewer views all of it, and whether server sends a 10 second packet, but client pauses at the 5s mark (which needs client-side logic)

Might sound trivial, but at netflix scale there's guaranteed a developer dedicated to that, probably a team, and maybe even a department.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#176

Post by PittleyDunkin »

nicce wrote: post #167
PittleyDunkin wrote: post #162

Earlier quoted context omitted.

> Whatever your service is, usually the database is the bottleneck. The database limits the latency, scaling and availability. Database-driven traffic is still a tiny percentage of internet traffic. It's harder to tell these days with encryption but on any given page-load on any project I've worked on, most of the traffic is in assets, not application data. Now, latency might be a different issue, but it seems ridicu…

> It's harder to tell these days with encryption but on any given page-load on any project I've worked on, most of the traffic is in assets, not application data. Yet you have to design the whole infrastructure to note that tiny margin to work flawlessly, because otherwise the service usually is not driving its purpose. Read-only assets are the easy part, which was my original claim.

> Read-only assets are the easy part, which was my original claim.

I don't think this is true at all given the volume. With that kind of scale everything is hard. It's just a different sort of hard than contended resources. Hell, even that is as "easy" these days with CRDTs (and I say this with dripping sarcasm).

Re: We built a self-healing system to survive a concurrency bug at Netflix

#178

Post by SnoozingBoa »

In technical context leaving unknowns with unknown boundaries behind is an approach that makes it hard to assess next actions in error cases, but even more importantly it makes the future system design work rooted on uncertainty.

To me that is blocker to my thinking. I really need to understand the impact of leaving something behind before continuing. I’d likely do everything in my power to remove that unknown from current state for the sake of sanity.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#179

Post by vishaldalwadi26 »

> Practical engineering can mean many things, but a definition I often return to is: having clear goals and making choices that are aligned with them.

The main takeaway from this post. You will not encounter exactly this or similar issues at your workplace, but this single piece of advice can help you fix any issues that you do encounter.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#180

Post by cnity »

ooFieTh6 wrote: post #80
cnity wrote: post #70

Earlier quoted context omitted.

Well, why? This comment seems counter to the now-popular "cattle not pets" approach.

state

It is pretty typical these days for services in a distributed architecture to not depend on local state whatsoever. In fact, in k8s there is no way to "properly reboot" a pod. The equivalent would be to replace the pod with a new one.
Post reply on HN