Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

code.facebook.com

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#11

Post by karavelov »

cperciva wrote: post #8

The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links. Not just connections which traverse aggregated links. If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets. What I don't understand is why Fac…

That was also my first idea about solving it. No idea why they didn't took this route.

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#12

Post by mcpherrinm »

donavanm wrote: post #10

Color me confused. Just a week ago they posted "We also have server-side means to “hash away” and route around trouble spots, if they occur." https://code.facebook.com/posts/360346274145943/introducing-... Does the server side flow hash manipulation dependent on ip tuple manipulation? Only enabled for some hosts/devices? A bit disappointed, was hoping for clever dscp or mpls tag manipulation. If they rely on new stre…

This problem has been ongoing for two years: the post you linked described their new data center design, which may not even be online yet.

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#13

Post by mcpherrinm »

cperciva wrote: post #8

The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links. Not just connections which traverse aggregated links. If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets. What I don't understand is why Fac…

But the cache is oblivious to which links the TCP connections are going over (well, except via latency & throughput as the post showed). If I understand what you propose, they need to add more bookkeeping to track what the connection did, instead of just doing pool.insert(conn_fd, time.now()). It sounds like the fix was just swapping the side you evict from, which could have been as little as a one line change.

Without a bit more information I think I am just speculating and don't really know. Their solution doesn't really taste bad to me though.

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#14

Post by ak217 »

donavanm wrote: post #9
fleitz wrote: post #4

Oh man, this reminds me of troubleshooting a host behind a router that didn't support window scaling. It took a week or two to figure out why we couldn't transfer data but could connect, ssh, ping, etc.

Broken TCP window scaling? The connection should still work, with the sender backing off right? If you mean busted PMTUD yeah thats awesome when the handshake and GET works, but you cant get data back.

We saw busted PMTUD on a customer's network trying to get to us via a misconfigured network link. It's a bad problem to debug. None of us knew what PMTUD was, or that TCP can require ICMP to work, until that day.

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#17

Post by ifdefdebug »

cperciva wrote: post #8

The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links. Not just connections which traverse aggregated links. If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets. What I don't understand is why Fac…

Out of nothing, I'd guess their MRU worked by just pushing an incoming free connection on top of a LIFO stack, while for the "most recent request sent" strategy you had to keep track of request timing and insert incoming free connections into the middle of the stack, complicating things.

Within their solution just had to convert the LIFO stack into an FIFO queue, keeping it simple.

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#18

Post by noselasd »

Many years ago I did programmeing on a telephony application which communicated using the old SS7 network. In here you, as a layer 4 protocol entity, can send a "link selection" key in the protocol messages. It's just a 4 bit number (or 8 if it's ANSI SS7 instead of ITU , iirc) per message, and the switches maps this number to a physical downstream link, often in a configurable fashion.

This is pretty neat, as the application can default to a round robin distribution and dynamically weigth the link selection key based on detected congestion/overload to shift the load to other links - since then I've often wished the TCP/IP world offered something similar.

Re: Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

#19

Post by stevejones »

cperciva wrote: post #8

The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links. Not just connections which traverse aggregated links. If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets. What I don't understand is why Fac…

I want to know why they're violating the end-to-end principle in their switches.
Post reply on HN