Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

101–110 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#101

Post by afavour »

Sl1mb0 wrote: post #94
louison11 wrote: post #85

You gotta pick your battles. Part of being in a startup is to be comfortable with quick and dirty when necessary. It’s when things get bigger, too corporate and slow that companies stop moving fast.

We are talking about Netflix. You know, the 'N' in FAANG/MAANG or whatever.

As a non-FAANGer Netflix has always intrigued me because of this. While Google, Facebook and others seem to have bogged themselves down in administrative mess, Netflix still seems agile. From the outside at least.

(also worth noting this post seems to be discussing an event that occurred many years ago, circa 2011, so might not be a reflection of where they are today)

Re: We built a self-healing system to survive a concurrency bug at Netflix

#102

Post by pmarreck »

Retric wrote: post #93
netdevnet wrote: post #54

Earlier quoted context omitted.

> It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months. I don't know why my senses tell me that this is wrong even if you can afford it

> I don't know why my senses tell me that this is wrong The fix is also hiding other issues that show up. So it degrades over time and eventually you’re stuck trying to solve multiple problems at the same time.

^ This is the problem. Not only that, solving 10 bugs (especially those more difficult nondeterministic concurrency bugs) at the same time is hideously harder than solving 1 at a time.

As a Director of Engineering at my last startup, I had an "all hands on deck" policy as soon as any concurrency bug was spotted. You do NOT want to let those fester. They are nondeterministic, infrequent, and exponentially dangerous as more and more appear and are swept under the rug via "reset-to-known-good" mitigations.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#103

Post by vladak »

xxs wrote: post #18

They were just lucky not to have data corruption due to concurrency issue, and the manifestation was infinite get. Overall if you can randomly "kill -9", the case is rather trivial. Likely replacing HashMap with CHM would not solve the concurrency issue either, but it'd prevent an infinite loop. (Edit) It appear that part is just wrong: "some calls to ConcurrentHashMap.get() seemed to be running infinitely." <-- it's…

yep, the link in the "some calls to ConcurrentHashMap.get() seemed to be running infinitely." sentence points to HashMap.html#get(java.lang.Object)

Re: We built a self-healing system to survive a concurrency bug at Netflix

#105

Post by pronoiac »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I guess this works right up until it doesn't? It's been a while, but I've seen AWS hit capacity for a specific instance size in a specific availability zone. I remember spot pricing being above the on-demand pricing, which might have been part of the issue.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#106

Post by tfandango »

braggerxyz wrote: post #92
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months. The trick is to not tell your manager that your bandaid works so well, but that it barely keeps the system alive and you need to introduce a proper fix. Been doing this for the last 10 years and we got our system so stable that I haven't had a midnight call in the last two years.

Classic trick. As a recent dev turned manager, these are the kind of things I've had a hard time learning.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#107

Post by l33t7332273 »

whatever1 wrote: post #62
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

Amazon needs wizards then.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#109

Post by nextos »

ken47 wrote: post #104

If the principles of languages like Erlang were taught in American school, things like this would be much likely to occur. Silly that Computer Science is regarded more highly by many than Software Engineering for Software Engineering jobs.

Ideas stemming from Erlang and Mozart/Oz are indeed a big blind spot in most undergrad programs. Sadly, even in EU all this is becoming a niche topic, which is weird as today's applications are more concurrent and data-intensive than ever.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#110

Post by braggerxyz »

ksd482 wrote: post #6

This was a nice short read. A simple (temporary) solution, yet a clever one. How was he managing the instances? Was he using kubernetes, or did he write some script to manage the auto terminating of the instances? It would also be nice to know why: 1. Killing was quicker than restarting. Perhaps because of the business logic built into the java application? 2. Killing was safe. How was the system architectured so tha…

> It would also be nice to know why:

1. Killing was quicker than restarting.

If you happen to restart one of the instances that was hanging in the infinite thread, you can wait a very long time until the Java container actually decides to kill itself because it did not finish its graceful shutdown within the alotted timeout period. Some Java containers have a default of 300s for this. In this circumstance kill -9 is faster by a lot ;)

Also we had circumstances where the affected Java container did not stop even if the timeout was reached because the misbehaving thread did consume the whole cpu and none was left for the supervisor thread. Then you can only kill the host process of the JVM.

Post reply on HN