Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

71–80 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#71

Post by cpursley »

conradfr wrote: post #28

I have a project where one function (reading metadata from an Icecast stream [0]) was causing a memory leak and ultimately consuming all of it. I don't remember all the details but I've still not be able to find the bug. But this being in Elixir I "fixed it" with Task, TaskSupervisor and try/catch/rescue. Not really a win but it is still running fine to this day. [0] https://github.com/conradfr/ProgRadio/blob/1fa12ca…

Half of hn posts are people showing off things where they spent a herculean amount of effort reinventing something that elixir/erlang has had solved 30+ years already.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#72

Post by jasonjayr »

rsynnott wrote: post #64
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> So much so that resolving the root cause isn't considered a priority and so we've had this running for months. I mean, you probably know this, but sooner or later this attitude is going to come back to bite you. What happens when you need to do it every hour? Every ten minutes? Every 30 seconds? This sort of solution is really only suitable for use as short-term life-support; unless you understand exactly what is h…

Well that's the thing: a bug that happens every 2 hrs and cannot be traced easily gives a developer roughly 4 opportunities in an 8hr day to reproduce + diagnose.

Once it's happening every 30 seconds, then they have up to 120 opportunities per hour, and it'll be fixed that much quicker!

Re: We built a self-healing system to survive a concurrency bug at Netflix

#73

Post by HL33tibCe7 »

bongodongobob wrote: post #29
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

"It's shockingly stable." You're running a soup. I'm not sure if this is satire or not. This reminds me of using a plug-in light timer to reboot your servers because some java program eats all the memory.

Sometimes running a soup is the correct decision

Re: We built a self-healing system to survive a concurrency bug at Netflix

#74

Post by zoky »

crabbone wrote: post #69
forkerenok wrote: post #43

Earlier quoted context omitted.

If you squint hard enough, this is an implementation of a higher order garbage collection: MarkNothingAndSweepEverything. There, formalized the approach, so you can't call it terrible anymore.

Oh no it isn't. Garbage collector needs to prove that what's being collected is garbage. If objects get collected because of an error... that's not really how you want GC to work. If you are looking for an apt metaphor, Stalin sort might be more in line with what's going on here. Or maybe "ostrich algorithm".

I think it’s more like Tech Support Sort, as in “Try turning it off and on again and see if it’s sorted”.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#75

Post by merizian »

This reminds me of LLM pretraining and how there are so many points at which the program could fail and so you need clever solutions to keep uptime high. And it's not possible to just fix the bugs--GPUs will often just crash (e.g. in graphics, if a pixel flips the wrong color for a frame, it's fine, whereas such things can cause numerical instability in deep learning so ECC catches them). You also often have a fixed sized cluster which you want to maximize utilization of.

So improving uptime involves holding out a set of GPUs to swap out failed ones while they reboot. But also the whole run can just randomly deadlock, so you might solve that by listening to the logs and restarting after a certain amount of inactivity. And you have to be clever with how to save/load checkpoints, since that can start to become a huge bottleneck.

After many layers of self healing, we managed to take a vacation for a few days without any calls :)

Re: We built a self-healing system to survive a concurrency bug at Netflix

#76

Post by ramchip »

girishso wrote: post #30

Not much familiar with Elixir OTP, but isn’t the approach OP took similar to Let It Crash philosophy of OTP?

Not really, you wouldn't normally kill or restart processes randomly in an OTP system. "Let it crash" is more about separating error handling from business logic.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#78

Post by znpy »

whatever1 wrote: post #62
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

Amazon folks won’t debug your code though, they’ll just happily bill you more.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#79

Post by DrBazza »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

Sounds like process-level garbage collection. Just kill it and restart. Which also sound like the apocryphal tale about the leaky code and the missile.

"This sparked and interesting memory for me. I was once working with a customer who was producing on-board software for a missile. In my analysis of the code, I pointed out that they had a number of problems with storage leaks. Imagine my surprise when the customers chief software engineer said "Of course it leaks"

He went on to point out that they had calculated the amount of memory the application would leak in the total possible flight time for the missile and then doubled that number. They added this much additional memory to the hardware to "support" the leaks. Since the missile will explode when it hits it's target or at the end of it's flight, the ultimate in garbage collection is performed without programmer intervention."

https://x.com/pomeranian99/status/858856994438094848

Re: We built a self-healing system to survive a concurrency bug at Netflix

#80

Post by ooFieTh6 »

cnity wrote: post #70
rullelito wrote: post #59

> Why not just reboot them? Terminating was faster. If you don't know why you should reboot servers/services properly instead of terminating them..

Well, why? This comment seems counter to the now-popular "cattle not pets" approach.

state
Post reply on HN