I like the practicality of this
We built a self-healing system to survive a concurrency bug at Netflix
11–20 of 184 posts
Re: We built a self-healing system to survive a concurrency bug at Netflix
#12One of the things I am greatful for kubernetes and the killing of pods. Had a similar problem but memory wise with a pesky memory leak, and the short term solution was to do nothing as instances would to do nothing.
During one of my past gigs, this exact feature hid a huge memory leak, in old code, always running on k8s which we found out only when we moved some instances to bare metal.
We hit this in a past gig too. One of the big services had a leak, but deployed every 24 hours which was hiding it. When the holiday deploy freeze hit the pods lived much longer than normal and caused an OOM storm.
At first I thought maybe we should add a "hack" to cycle all the pods over 24 hours old, but then I wondered if making holiday freezes behave like normal weeks was really a hack at all or just reasonable predictability.
In the end folks managed to fix the leak and we didn't resolve the philosophical question though.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#13Reminds me of the famous quote by Rasmus Lerdorf, creator of PHP
> I’m not a real programmer. I throw together things until it works then I move on. The real programmers will say “Yeah it works but you’re leaking memory everywhere. Perhaps we should fix that.” I’ll just restart Apache every 10 requests.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#14> and to my memory, some calls to ConcurrentHashMap.get() seemed to be running infinitely.
Of course they did. And whoever though "Concurrent" meant it would work fine gets burned by it. Of course.
And of course it doesn't work properly or intuitively for some very stupid reason. Sigh
Re: We built a self-healing system to survive a concurrency bug at Netflix
#15Reminds me of the famous quote by Rasmus Lerdorf, creator of PHP > I’m not a real programmer. I throw together things until it works then I move on. The real programmers will say “Yeah it works but you’re leaking memory everywhere. Perhaps we should fix that.” I’ll just restart Apache every 10 requests.
i ll argue that doing the restart is more important until someone else finds the leak
Re: We built a self-healing system to survive a concurrency bug at Netflix
#16My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions.
Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically.
Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.
It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#17Reminds me of the famous quote by Rasmus Lerdorf, creator of PHP > I’m not a real programmer. I throw together things until it works then I move on. The real programmers will say “Yeah it works but you’re leaking memory everywhere. Perhaps we should fix that.” I’ll just restart Apache every 10 requests.
i ll argue that doing the restart is more important until someone else finds the leak
Or future me. It hurts on the inside to just kick EC2 every hour because every 61 minutes something goes awry in the process. But the show must go on, so you put in the temporary fix knowing that it's not going to be temporary. Still, weeks/months/years down the line you could get lucky and the problem will go away and you can remove the kludge. But if you're ridiculously lucky, not only will the problem get fixed, but you'll get to understand exactly why the mysterious problem was happening in the first place. Like the gunicorn 500 upgrade bug, or the Postgres TOAST json thing. That sort of satisfaction isn't something money can buy. (Though it will help pay for servers in the interim until you find the bug.)
Re: We built a self-healing system to survive a concurrency bug at Netflix
#18They were just lucky not to have data corruption due to concurrency issue, and the manifestation was infinite get. Overall if you can randomly "kill -9", the case is rather trivial.
Likely replacing HashMap with CHM would not solve the concurrency issue either, but it'd prevent an infinite loop. (Edit) It appear that part is just wrong: "some calls to ConcurrentHashMap.get() seemed to be running infinitely." <-- it's possible to happen no hashmap during concurrent put(s), but not to ConcurrentHashMap
Re: We built a self-healing system to survive a concurrency bug at Netflix
#19My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)
Re: We built a self-healing system to survive a concurrency bug at Netflix
#20> and to my memory, some calls to ConcurrentHashMap.get() seemed to be running infinitely. Of course they did. And whoever though "Concurrent" meant it would work fine gets burned by it. Of course. And of course it doesn't work properly or intuitively for some very stupid reason. Sigh
It has to be an error - it could happen to HashMap, it has never been an issue w/ CHM.