Metastable Failures in Distributed Systems
1–10 of 12 posts
Re: Metastable Failures in Distributed Systems
#2The HotOS'21 paper (Aleksey is one of the authors) is very much worth reading if you're building or maintaining large-scale distributed systems: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...
I wrote a blog post about this paper, too: https://brooker.co.za/blog/2021/05/24/metastable.html
Re: Metastable Failures in Distributed Systems
#3I had never seen the term “goodput” before. Kind of a clumsy portmanteau...
Re: Metastable Failures in Distributed Systems
#4Interesting article. It seems like if you had some kind of generic load shedding component that could monitor the difference between goodput and throughput and throttle load until the former increases it might help? Of course how to do this while guaranteeing it won't make the situation worse is another thing.
Re: Metastable Failures in Distributed Systems
#5The HotOS'21 paper (Aleksey is one of the authors) is very much worth reading if you're building or maintaining large-scale distributed systems: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s... I wrote a blog post about this paper, too: https://brooker.co.za/blog/2021/05/24/metastable.html
While reading Aleksey’s blog[0] linked from yours it struck me that this sounds a great deal like (real world) traffic engineering. A backup occurs, the traffic grinds to a halt, and after the blockage is removed it takes a very long time for the system to recover. If traffic is heavy enough, even just a small slowdown is enough to cause prolonged pain.
[0]: http://charap.co/metastable-failures-in-distributed-systems/
(Having said that, my traffic engineering cluefulness is roughly at the “casual reader glancing at headlines” level, if that, but I found the comparison interesting.)
I’m unfamiliar with HotOS, so I’m curious about the context behind this sentence:
> But it's a HotOS paper, and this sure is a hot topic, so I won't hold the lack of a rigorous investigation of the background against the authors.
Re: Metastable Failures in Distributed Systems
#6> Fast paths, caches, retries, failover, load balancing, and autoscaling all make the failure state less resource-efficient relative to the normal state, which makes the feedback loop worse. Beware of very high cache hit rates!
This scares me as someone who is working on performance tuning and is adding fast paths every other day....
The paper kinds of reminded me the "Gray failure" paper by the way.
https://blog.acolyer.org/2017/06/15/gray-failure-the-achille...
Re: Metastable Failures in Distributed Systems
#7One is reminded of Nassim Taleb's writing on efficiency and fragility. Long-lived systems (e.g. in biology) have a built in surplus, rather than operating everything at peak efficiency, because longevity is often determined not by how well you perform normally, but how well you perform during your worst crisis moments.
Re: Metastable Failures in Distributed Systems
#8I had never seen the term “goodput” before. Kind of a clumsy portmanteau... https://en.m.wikipedia.org/wiki/Goodput
In other contexts I’ve seen its opposite called “failure demand”, which I think is an apt notion for this context as well: Increased load on the system, not from original users’ intent, but from the response to failures, including retries from the user due to failures of prior interactions.
I found the paper on metastable failures brought back a lot of systems theory for me.
Re: Metastable Failures in Distributed Systems
#9Here’s a simple one from my career. A calls B calls C. C makes a bad deploy. B passes through the errors from C and A retries against them. The retries push B’s utilization past the limit and instances start getting OOM killed. The C deploy is reverted. But the retries from A to B aren’t about C anymore, they’re about the fact that B is overloaded. A and B are in a neat little death spiral. We have to stop A, let B recover, and then slowly turn A back on.
After this incident we installed a per-instance rate limit on all services.
Re: Metastable Failures in Distributed Systems
#10One is reminded of Nassim Taleb's writing on efficiency and fragility. Long-lived systems (e.g. in biology) have a built in surplus, rather than operating everything at peak efficiency, because longevity is often determined not by how well you perform normally, but how well you perform during your worst crisis moments.
I'm certainly reminded of the current semiconductor shortages, not to mention early-2020 toilit paper shortages. People see low stock of an essential product, buy up what they need and some spare to cover an anticipated shortage, and the stock runs out, causing a shortage. Other people then try to buy whaxever they can to build their own buffers, extending the shortages.