Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
1–10 of 40 posts
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#2This is one of the reasons why I like to include debugging a problem with a couple of levels of indirection for in-depth technical interviews.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#3This was a great investigation of what must surely have been a complicated, frustrating, and expensive problem. Awesome writeup.
I love reading about post-mortems like this, even if they're unlikely to happen at my startup, because the problem-solving techniques that get displayed tend to generalize to things of almost any size.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#4Oh man, this reminds me of troubleshooting a host behind a router that didn't support window scaling.
It took a week or two to figure out why we couldn't transfer data but could connect, ssh, ping, etc.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#5"The breakthrough came when we started thinking about the components in the system as malicious actors colluding via covert channels."
That's probably the piece of that article that I'm going to remember for a long time.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#6" At a meta-level, the next time you are debugging emergent behavior, you might try thinking of the components as agents colluding via covert channels."
An insightful way of looking at a problem when the cause is not obvious.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#7"The breakthrough came when we started thinking about the components in the system as malicious actors colluding via covert channels." That's probably the piece of that article that I'm going to remember for a long time.
That's exactly the thinking which led me to the first published cryptographic side channel attack against hyperthreading: Intel's optimization manual had a comment about being careful with stack alignment to avoid poor cache performance, and I thought "what if slow could instead be maliciously slow?"
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#8The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links.
Not just connections which traverse aggregated links. If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets.
What I don't understand is why Facebook didn't simply fix their MRU pool: Switching from "most recent response received" to "most recent request sent" (out of the links which don't have a request already in progress, of course) would have flipped the effect from preferring overloaded links to avoiding them.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#9Oh man, this reminds me of troubleshooting a host behind a router that didn't support window scaling. It took a week or two to figure out why we couldn't transfer data but could connect, ssh, ping, etc.
Broken TCP window scaling? The connection should still work, with the sender backing off right? If you mean busted PMTUD yeah thats awesome when the handshake and GET works, but you cant get data back.
Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
#10Color me confused. Just a week ago they posted "We also have server-side means to “hash away” and route around trouble spots, if they occur."
https://code.facebook.com/posts/360346274145943/introducing-...
Does the server side flow hash manipulation dependent on ip tuple manipulation? Only enabled for some hosts/devices? A bit disappointed, was hoping for clever dscp or mpls tag manipulation. If they rely on new streams with specific src ports its a lot less interesting, and less useful for long lived connections.