Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

81–90 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#81

Post by fidotron »

This is a bit odd coming from the company of chaos engineering - has the chaos monkey been abandoned at Netflix?

I have long advocated randomly restarting things with different thresholds partly for reasons like this* and to ensure people are not complacent wrt architecture choices. The resistance, which you can see elsewhere here, is huge, but at scale it will happen regardless of how clever you try to be. (A lesson from the erlang people that is often overlooked).

* Many moons ago I worked on a video player which had a low level resource leak in some decoder dependency. Luckily the leak was attached to the process, so it was a simple matter of cycling the process every 5 minutes and seamlessly attaching a new one. That just kept going for months on end, and eventually the dependency vendor fixed the leak, but many years later.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#82

Post by chronid »

whatever1 wrote: post #62
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

This has been done forever. Ops team had cronjobs to restart misbehaving applications out of business hours since before I started working. In a previous job, the solution for disks being full on a VM on-prem (no, not databases) was an automatic reimage. I've seen scheduled index rebuilds on Oracle. The list goes on.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#83

Post by nukethegrbj »

crabbone wrote: post #69
forkerenok wrote: post #43

Earlier quoted context omitted.

If you squint hard enough, this is an implementation of a higher order garbage collection: MarkNothingAndSweepEverything. There, formalized the approach, so you can't call it terrible anymore.

Oh no it isn't. Garbage collector needs to prove that what's being collected is garbage. If objects get collected because of an error... that's not really how you want GC to work. If you are looking for an apt metaphor, Stalin sort might be more in line with what's going on here. Or maybe "ostrich algorithm".

>Garbage collector needs to prove that what's being collected is garbage

Some collectors may need to do this, but there are several collectors that don't. EpsilonGC is a prime example of a GC that doesen't need to prove anything

Re: We built a self-healing system to survive a concurrency bug at Netflix

#84

Post by ricardobeat »

fidotron wrote: post #81

This is a bit odd coming from the company of chaos engineering - has the chaos monkey been abandoned at Netflix? I have long advocated randomly restarting things with different thresholds partly for reasons like this* and to ensure people are not complacent wrt architecture choices. The resistance, which you can see elsewhere here, is huge, but at scale it will happen regardless of how clever you try to be. (A lesson…

In cases like this won't Chaos Monkey actually hide the problem, since it's basically doing exactly the same as their mitigation strategy - randomly restarting services?

Re: We built a self-healing system to survive a concurrency bug at Netflix

#86

Post by alecco »

Self-healing system: increase cluster size and replace servers randomly. It works because it was a problem of threads occasionally entering an infinite loop but not corrupting data. And the whole system can tolerate these kind of whole server crashes. IMHO an unusual combination of preconditions.

It's not explained why they couldn't write a monitor script instead to find servers having the issue and only killing those.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#88

Post by fidotron »

ricardobeat wrote: post #84
fidotron wrote: post #81

This is a bit odd coming from the company of chaos engineering - has the chaos monkey been abandoned at Netflix? I have long advocated randomly restarting things with different thresholds partly for reasons like this* and to ensure people are not complacent wrt architecture choices. The resistance, which you can see elsewhere here, is huge, but at scale it will happen regardless of how clever you try to be. (A lesson…

In cases like this won't Chaos Monkey actually hide the problem, since it's basically doing exactly the same as their mitigation strategy - randomly restarting services?

Right. The point of the question is why not ramp up the monkey? They seem to imply it isn’t there now, which wouldn’t surprise me with the cultural shifts that have occurred in the tech world.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#90

Post by snicker7 »

znpy wrote: post #78
whatever1 wrote: post #62

Earlier quoted context omitted.

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

Amazon folks won’t debug your code though, they’ll just happily bill you more.

The point is not to spend time frantically fixing code at 3 AM.
Post reply on HN