Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#121

Post by 4star3star »

alecco wrote: post #86

Self-healing system: increase cluster size and replace servers randomly. It works because it was a problem of threads occasionally entering an infinite loop but not corrupting data. And the whole system can tolerate these kind of whole server crashes. IMHO an unusual combination of preconditions. It's not explained why they couldn't write a monitor script instead to find servers having the issue and only killing thos…

I think they just needed a quick and dirty solution that was good enough for a few days. They figured that for 1% failure per hour, they needed to kill x processes every y minutes to keep ahead of the failures. I'm sure it would be much more efficient but also more complicated to try to target the specific failures, and the "good enough" solution was acceptable.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#122

Post by mhawthorne »

vladak wrote: post #103
xxs wrote: post #18

They were just lucky not to have data corruption due to concurrency issue, and the manifestation was infinite get. Overall if you can randomly "kill -9", the case is rather trivial. Likely replacing HashMap with CHM would not solve the concurrency issue either, but it'd prevent an infinite loop. (Edit) It appear that part is just wrong: "some calls to ConcurrentHashMap.get() seemed to be running infinitely." <-- it's…

yep, the link in the "some calls to ConcurrentHashMap.get() seemed to be running infinitely." sentence points to HashMap.html#get(java.lang.Object)

[dead]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#123

Post by rakoo »

JSDevOps wrote: post #25
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

This sounds terrible

To be fair this is what the BEAM vm structures everything on: If something is wonky, crash it and restart from a known ok state. Except when BEAM does it everyone says it's brilliant

Re: We built a self-healing system to survive a concurrency bug at Netflix

#124

Post by mhawthorne »

xxs wrote: post #20
raverbashing wrote: post #14

> and to my memory, some calls to ConcurrentHashMap.get() seemed to be running infinitely. Of course they did. And whoever though "Concurrent" meant it would work fine gets burned by it. Of course. And of course it doesn't work properly or intuitively for some very stupid reason. Sigh

It has to be an error - it could happen to HashMap, it has never been an issue w/ CHM.

[dead]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#125

Post by Jnr »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I am running an old statically compiled perl binary that has a memory leak. So every day the container is restarted automatically so I would not have to deal with the problem. It has been running like this for many many years now.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#126

Post by posix_compliant »

What's neat is that this is a differential equation. If you kill 5% of instances each hour, the reduction in bad instances is proportional to the current number of instances.

i.e.

if bad(t) = fraction of bad instances at time t

and

bad(0) = 0

then

d(bad(t))/dt = -0.05 * bad(t) + 0.01 * (1 - bad(t))

so

bad(t) = 0.166667 - 0.166667 e^(-0.06 t)

Which looks a mighty lot like the graph of bad instances in the blog post.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#127

Post by Gud »

whatever1 wrote: post #62
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

This is not exactly a new tactic, and not something that would have to have been implemented without any cloud solution. A randomized 'kill -HUP' could do the same thing, for example.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#128

Post by eigenvalue »

I understand how their approach worked well enough, but I don’t get why they couldn’t selectively target the VMs that were currently experiencing problems rather than randomly select any VM to terminate. If they were exhausting all their CPU resources, wouldn’t that be easy enough to search for using something like ansible?

Re: We built a self-healing system to survive a concurrency bug at Netflix

#129

Post by mhawthorne »

pronoiac wrote: post #105
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I guess this works right up until it doesn't? It's been a while, but I've seen AWS hit capacity for a specific instance size in a specific availability zone. I remember spot pricing being above the on-demand pricing, which might have been part of the issue.

[dead]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#130

Post by toast0 »

TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

> But as time goes by I just ask, all this work and costs and complexity, to serve files?

IMHO, a large amount of the complexity is all the other stuff. Account information, browsing movies, recommendations, viewed/not/how much seen, steering to local CDN nodes, DRM stuff, etc.

The file servers have a lot less complexity; copy content to CDN nodes, send the client to the right node for the content, serve 400Gbps+ per node. Probably some really interesting stuff for their real time streams (but I haven't seen a blog/presentation on those)

Transcoding is probably interesting too. Managing job queues isn't new, but there's probably some fun stuff around cost effectiveness.

Post reply on HN