We built a self-healing system to survive a concurrency bug at Netflix

Post by hughesjj » Thu, Nov 14, 2024, 9:38 AM UTC

pronoiac wrote: post #105

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I guess this works right up until it doesn't? It's been a while, but I've seen AWS hit capacity for a specific instance size in a specific availability zone. I remember spot pricing being above the on-demand pricing, which might have been part of the issue.

Post by nicce » Thu, Nov 14, 2024, 7:38 PM UTC

PittleyDunkin wrote: post #176

nicce wrote: post #167

Earlier quoted context omitted.

> It's harder to tell these days with encryption but on any given page-load on any project I've worked on, most of the traffic is in assets, not application data. Yet you have to design the whole infrastructure to note that tiny margin to work flawlessly, because otherwise the service usually is not driving its purpose. Read-only assets are the easy part, which was my original claim.

> Read-only assets are the easy part, which was my original claim. I don't think this is true at all given the volume. With that kind of scale everything is hard. It's just a different sort of hard than contended resources. Hell, even that is as "easy" these days with CRDTs (and I say this with dripping sarcasm).