Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

21–30 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#21

Post by bratbag »

gorkempacaci wrote: post #19
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)

Depends what else it's solving for.

I've seen multiple issues solved like this after engineering teams have been cut to the bone.

If the cost of maintaining enough engineers to keep systems stable for more than 24 hours, is more than the cost of doubling the container count, then this is what happens

Re: We built a self-healing system to survive a concurrency bug at Netflix

#22

Post by kenhwang »

gorkempacaci wrote: post #19
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)

Depends on how long it takes for the incoming instances to initialize and outgoing instances to fully decommission.

x = time it takes to switchover

y = length of the cycles

x/y = % increase in cost

For us, it's 15 minutes / 120 minutes = 12.5% increase, which was deemed acceptable enough for a small service.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#23

Post by nicman23 »

fragmede wrote: post #17
nicman23 wrote: post #15

Earlier quoted context omitted.

i ll argue that doing the restart is more important until someone else finds the leak

Or future me. It hurts on the inside to just kick EC2 every hour because every 61 minutes something goes awry in the process. But the show must go on, so you put in the temporary fix knowing that it's not going to be temporary. Still, weeks/months/years down the line you could get lucky and the problem will go away and you can remove the kludge. But if you're ridiculously lucky, not only will the problem get fixed, b…

or at least after the weekend :P

Re: We built a self-healing system to survive a concurrency bug at Netflix

#24

Post by toast0 »

gorkempacaci wrote: post #19
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)

Shouldn't be too high cost if you only run 2x the instances for a short amount of time. A reasonable use of Cloud, IMHO, if you can't figure out a less disruptive bandaid.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#25

Post by JSDevOps »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

This sounds terrible

Re: We built a self-healing system to survive a concurrency bug at Netflix

#26

Post by JSDevOps »

bratbag wrote: post #21
gorkempacaci wrote: post #19

Earlier quoted context omitted.

How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)

Depends what else it's solving for. I've seen multiple issues solved like this after engineering teams have been cut to the bone. If the cost of maintaining enough engineers to keep systems stable for more than 24 hours, is more than the cost of doubling the container count, then this is what happens

This. All the domain knowledge has left. This sounds like a Hacky work around at best which AWS will welcome you with open arms come invoice day.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#27

Post by anal_reactor »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

I've realized that majority of engineers have no critical thinking, and are unable to see things beyond their domain of speciality. Arguments like "even when accounting for potential incident, your solution is more expensive, while our main goal is making money" almost never work, and I've been in countless discussions where some random document with "best practices", whatever they are supposed to be, was treated like a sacred scripture.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#28

Post by conradfr »

I have a project where one function (reading metadata from an Icecast stream [0]) was causing a memory leak and ultimately consuming all of it.

I don't remember all the details but I've still not be able to find the bug.

But this being in Elixir I "fixed it" with Task, TaskSupervisor and try/catch/rescue.

Not really a win but it is still running fine to this day.

[0] https://github.com/conradfr/ProgRadio/blob/1fa12ca73a40aedb9...

Re: We built a self-healing system to survive a concurrency bug at Netflix

#29

Post by bongodongobob »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

"It's shockingly stable." You're running a soup. I'm not sure if this is satire or not. This reminds me of using a plug-in light timer to reboot your servers because some java program eats all the memory.
Post reply on HN