I mean, it worked for Boeing[1] too.
1 - https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
We built a self-healing system to survive a concurrency bug at Netflix
31–40 of 184 posts
Re: We built a self-healing system to survive a concurrency bug at Netflix
#32My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
I've realized that majority of engineers have no critical thinking, and are unable to see things beyond their domain of speciality. Arguments like "even when accounting for potential incident, your solution is more expensive, while our main goal is making money" almost never work, and I've been in countless discussions where some random document with "best practices", whatever they are supposed to be, was treated lik…
We are dogmatic and emotional, but the temptation to base your opinions on the "deeper theory" is large.
Pragmatically, restart the service periodically and spend your time on more pressing matters.
On the other hand, we fully understand the reason for the fault, but we don't know exactly where the fault is. And it is, our fault. It takes a certain kind of discipline to say "there are many things I understand but don't have the time to master now, let's leave it."
It's, mostly, embarrassing.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#33My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
I've realized that majority of engineers have no critical thinking, and are unable to see things beyond their domain of speciality. Arguments like "even when accounting for potential incident, your solution is more expensive, while our main goal is making money" almost never work, and I've been in countless discussions where some random document with "best practices", whatever they are supposed to be, was treated lik…
that's because the judge(s) and executioner(s) aren't engineers, and the jury is not of their peers. and for the record i have a hard time faulting the non-engineers above so-described... they are just grasping for things they can understand and have input on. who wouldn't want that? it's not at all reasonable for the keepers of the pursestrings to expect a certain amount of genuflection by way of self-justification. no one watches the watchers... but they're the ones watching, so may as well present them with a verisimilitudinous rendition of reality... right?
but, as a discipline, engineers manage to encourage the ascent of the least engineer-ly (or, perhaps, "hacker"-ly) among them ("us") ...-selves... through their sui generis combination of learned helplessness, willful ignorance, incorrigible myopia, innate naïvete, and cynical self-servitude that signify the Institutional (Software) Engineer. coddled more than any other specialty within "the enterprise", they manage to simultaneously underplay their hand with respect to True Leverage (read: "Power") and overplay their hand with respect to complices of superiority. i am ashamed and dismayed to recall the numerous times i have heard (and heard of) comments to the effect of "my time is too expensive for this meeting" in the workplace... every single one of which has come not from the managerial class-- as one might reasonably, if superficially, expect-- but from the software engineer rank and file.
to be clear: i don't think it's fair to expect high-minded idealism from anyone. but if you are looking for the archetypical "company person"... engineers need look no further than their fellow podmates / slack-room-mates / etc. and thus no one should be surprised to see the state of the world we all collectively hath wrought.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#34Earlier quoted context omitted.
I've realized that majority of engineers have no critical thinking, and are unable to see things beyond their domain of speciality. Arguments like "even when accounting for potential incident, your solution is more expensive, while our main goal is making money" almost never work, and I've been in countless discussions where some random document with "best practices", whatever they are supposed to be, was treated lik…
We are dogmatic and emotional, but the temptation to base your opinions on the "deeper theory" is large. Pragmatically, restart the service periodically and spend your time on more pressing matters. On the other hand, we fully understand the reason for the fault, but we don't know exactly where the fault is. And it is, our fault. It takes a certain kind of discipline to say "there are many things I understand but don…
"certain kind" of discipline, indeed... not the good kind. and while your comment goes to great pains to highlight how that particular God is dead (and i agree, for the record), the God of Quality (the one that Pirsig goes to great lengths to not really define) toward which the engineer's heart of heart prays that lives within us all is... unimpressed, to say the least.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#35My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
"It's shockingly stable." You're running a soup. I'm not sure if this is satire or not. This reminds me of using a plug-in light timer to reboot your servers because some java program eats all the memory.
or installing software to jiggle the mouse every so often so that the computer with the spreadsheet that runs the company doesn't go to sleep
Re: We built a self-healing system to survive a concurrency bug at Netflix
#36I mean, it worked for Boeing[1] too. 1 - https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
"worked"
(not that i don't get the sarcasm)
Re: We built a self-healing system to survive a concurrency bug at Netflix
#37> and to my memory, some calls to ConcurrentHashMap.get() seemed to be running infinitely. Of course they did. And whoever though "Concurrent" meant it would work fine gets burned by it. Of course. And of course it doesn't work properly or intuitively for some very stupid reason. Sigh
It has to be an error - it could happen to HashMap, it has never been an issue w/ CHM.
this sounds more like citing chapter and verse in an exegesis than anything of direct relevance to the Mortal Plane...
Re: We built a self-healing system to survive a concurrency bug at Netflix
#38I mean, it worked for Boeing[1] too. 1 - https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
True, but for somewhat different reasons. For the OP, they take this approach because they simply don't know yet what the problem is, and it would take some time to track it down and fix it and they don't want to bother.
For Boeing, it's probably something fairly simple actually, but they don't want to fix it because their software has to go through a strict development process based on requirements and needing certification and testing, so fixing even a trivial bug is extremely time-consuming and expensive, so it's easier to just put a directive in the manual saying the equipment needs to be power-cycled every so often and let the users deal with it. The OP isn't dealing with this kind of situation.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#39Yeah I had the same issue of my EC2 that I used to host my personal websites randomly getting to 100% CPU and being unreachable.
I put a CloudWatch alarm at 90% CPU usage which would trigger a reboot (which completed way before anyone would notice a downtime).
Never had issues again.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#40Interesting read, the fix seems to be straightforward, but I'd have a few more questions if I was trying to do something similar. Is software deployed regularly on this cluster? Does that deployment happen faster than the rate at which they were losing CPUs? Why not just periodically force a deployment, given it's a repeated process that probably already happens frequently. What happens to the clients trying to conne…
An answer to basically all your questions is: doesn’t matter, they did their best to stabilize in a short amount of time, and it worked - that’s what mattered.