41–50 of 184 posts
That was a bit underwhelming compared to what the headline set my expectations up for, but definitely a good idea and neat solution.
Earlier quoted context omitted.
We are dogmatic and emotional, but the temptation to base your opinions on the "deeper theory" is large. Pragmatically, restart the service periodically and spend your time on more pressing matters. On the other hand, we fully understand the reason for the fault, but we don't know exactly where the fault is. And it is, our fault. It takes a certain kind of discipline to say "there are many things I understand but don…
"certain kind" of discipline, indeed... not the good kind. and while your comment goes to great pains to highlight how that particular God is dead (and i agree, for the record), the God of Quality (the one that Pirsig goes to great lengths to not really define) toward which the engineer's heart of heart prays that lives within us all is... unimpressed, to say the least.
Then you realize it's a paper idol and the best you can do is suck less than the average.
Thanks for playing Wing Commander!
My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
This sounds terrible
There, formalized the approach, so you can't call it terrible anymore.
My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
This sounds terrible
Earlier quoted context omitted.
We are dogmatic and emotional, but the temptation to base your opinions on the "deeper theory" is large. Pragmatically, restart the service periodically and spend your time on more pressing matters. On the other hand, we fully understand the reason for the fault, but we don't know exactly where the fault is. And it is, our fault. It takes a certain kind of discipline to say "there are many things I understand but don…
"certain kind" of discipline, indeed... not the good kind. and while your comment goes to great pains to highlight how that particular God is dead (and i agree, for the record), the God of Quality (the one that Pirsig goes to great lengths to not really define) toward which the engineer's heart of heart prays that lives within us all is... unimpressed, to say the least.
Not OP but this is a somewhat normal case of making a tradeoff? They aren't able to repair it at the moment (or rather don't want/can't allocate the time for it) and instead trade their ressource usage for stability and technical debt.
Earlier quoted context omitted.
How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)
Shouldn't be too high cost if you only run 2x the instances for a short amount of time. A reasonable use of Cloud, IMHO, if you can't figure out a less disruptive bandaid.
Kill and restart the service. This seems to be the coder solution to everything. We do it for our service as well. The programmer could fix their stuff but alas, that’s too much to ask.
Memory leaks are often "resolved" this way... until time allows for a proper fix.
Earlier quoted context omitted.
Shouldn't be too high cost if you only run 2x the instances for a short amount of time. A reasonable use of Cloud, IMHO, if you can't figure out a less disruptive bandaid.
AWS charges instances in 1 hour increments - so you're paying 150% the EC2 costs if you're doing this every 2 hours
That was a bit underwhelming compared to what the headline set my expectations up for, but definitely a good idea and neat solution.
from the headline alone I got linkedin ceo vibe. "Built a Self-Healing System to Survive a Concurrency Bug" is how I could describe wrapping a failing method in a retry loop