Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

61–70 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#61

Post by ay »

xxs wrote: post #18

They were just lucky not to have data corruption due to concurrency issue, and the manifestation was infinite get. Overall if you can randomly "kill -9", the case is rather trivial. Likely replacing HashMap with CHM would not solve the concurrency issue either, but it'd prevent an infinite loop. (Edit) It appear that part is just wrong: "some calls to ConcurrentHashMap.get() seemed to be running infinitely." <-- it's…

it wasn’t luck, it was very deliberately engineered for. The article does lack a good bit of context about the Netflix infra:

https://netflixtechblog.com/the-netflix-simian-army-16e57fba...

https://github.com/Netflix/chaosmonkey

Re: We built a self-healing system to survive a concurrency bug at Netflix

#62

Post by whatever1 »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I think this is a prime example of why the cloud won.

You don’t need wizards in your team anymore.

Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#63

Post by kenhwang »

cryptonym wrote: post #60
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

People will argue you should spend time on something else once you put bandaid on a wooden leg. You should do proper risk assessment, such bug may be leveraged by an attacker, that may actually be a symptom of a running attack. That may also lead to data corruption or exposure. That may mean some part of the system are poorly optimised and over-consuming resources, maybe impacting user-experience. With a dirty workar…

It's just yet another piece of debt that gets prioritized against other pieces of debt. As long as the cost of this debt is purely fiscal, it's easy enough to position in the debt backlog. Maybe a future piece of debt will increase the cost of this. Maybe paying off another piece of debt will also pay off some of this. The tech debt payoff prioritization process will get to it when it gets to it.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#64

Post by rsynnott »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> So much so that resolving the root cause isn't considered a priority and so we've had this running for months.

I mean, you probably know this, but sooner or later this attitude is going to come back to bite you. What happens when you need to do it every hour? Every ten minutes? Every 30 seconds?

This sort of solution is really only suitable for use as short-term life-support; unless you understand exactly what is happening (but for some reason have chosen not to fix it), it's very, very dangerous.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#65

Post by cryptonym »

kenhwang wrote: post #63
cryptonym wrote: post #60

Earlier quoted context omitted.

People will argue you should spend time on something else once you put bandaid on a wooden leg. You should do proper risk assessment, such bug may be leveraged by an attacker, that may actually be a symptom of a running attack. That may also lead to data corruption or exposure. That may mean some part of the system are poorly optimised and over-consuming resources, maybe impacting user-experience. With a dirty workar…

It's just yet another piece of debt that gets prioritized against other pieces of debt. As long as the cost of this debt is purely fiscal, it's easy enough to position in the debt backlog. Maybe a future piece of debt will increase the cost of this. Maybe paying off another piece of debt will also pay off some of this. The tech debt payoff prioritization process will get to it when it gets to it.

Without proper risk assessment, that's poor management and a recipe for disaster. Without that assessment, you don't know the "cost", if that can even be measured. Of course one can still run a business without doing such risk assessment and poorly managing technical debt, just be prepared for higher disaster chances.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#66

Post by actionfromafar »

rsynnott wrote: post #64
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> So much so that resolving the root cause isn't considered a priority and so we've had this running for months. I mean, you probably know this, but sooner or later this attitude is going to come back to bite you. What happens when you need to do it every hour? Every ten minutes? Every 30 seconds? This sort of solution is really only suitable for use as short-term life-support; unless you understand exactly what is h…

In a way, yes. But it's also like a sledge hammer approach to stateless design. New code will be built within the constraint that stuff will be rebooted fairly often. That's not only a bad thing.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#67

Post by rothron »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

This fix means that you won't notice when you accumulate other such resource leaks. When the shit eventually hits the fan, you'll have to deal with problems you didn't even knew you had.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#68

Post by crabbone »

netdevnet wrote: post #54
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months. I don't know why my senses tell me that this is wrong even if you can afford it

Guys might be looking to match the fame of the SolarWinds.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#69

Post by crabbone »

forkerenok wrote: post #43
JSDevOps wrote: post #25

Earlier quoted context omitted.

This sounds terrible

If you squint hard enough, this is an implementation of a higher order garbage collection: MarkNothingAndSweepEverything. There, formalized the approach, so you can't call it terrible anymore.

Oh no it isn't. Garbage collector needs to prove that what's being collected is garbage. If objects get collected because of an error... that's not really how you want GC to work.

If you are looking for an apt metaphor, Stalin sort might be more in line with what's going on here. Or maybe "ostrich algorithm".

Re: We built a self-healing system to survive a concurrency bug at Netflix

#70

Post by cnity »

rullelito wrote: post #59

> Why not just reboot them? Terminating was faster. If you don't know why you should reboot servers/services properly instead of terminating them..

Well, why? This comment seems counter to the now-popular "cattle not pets" approach.
Post reply on HN