A very simple system is sufficient to demonstrate such metastable states. Consider a frontend supported by a database. Requests to the FE come in at a uniform rate below the ability of the database. Now there’s a peak in incoming requests. As parallel queries pile up, they start stepping on each other’s toes and the response time goes up. Given the temporarily stable incoming request rate, this means even more parallel queries and even slower response rate. Eventually the database response rate drops to the incoming request rate. If the peak of traffic subsides, the system sees a stable, usual, incoming request rate but a massively struggling database with a large number of parallel requests. This is the metastable state. When the db is restarted, purging the connections, everything returns to normal. Which can lead to the conclusion of the db platform being at fault. This is why you have queues or connection pools in front of your BE: to prevent dynamic complexity leaking over component boundaries and wreaking havoc all over the system.
Metastable Failures in Distributed Systems
11–12 of 12 posts
Re: Metastable Failures in Distributed Systems
#12The HotOS'21 paper (Aleksey is one of the authors) is very much worth reading if you're building or maintaining large-scale distributed systems: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s... I wrote a blog post about this paper, too: https://brooker.co.za/blog/2021/05/24/metastable.html
While reading Aleksey’s blog[0] linked from yours it struck me that this sounds a great deal like (real world) traffic engineering. A backup occurs, the traffic grinds to a halt, and after the blockage is removed it takes a very long time for the system to recover. If traffic is heavy enough, even just a small slowdown is enough to cause prolonged pain. [0]: http://charap.co/metastable-failures-in-distributed-systems…
HotOS is not considered to have the same rigor as other more prestigious journals. It’s often used for preliminary results or viewpoint papers, IIYUC.