Skype brief post-mortem

So the bug is in Skype client code that is responsible for handling supernode behavior.

The fix should probably be about developing deploying new version of Skype client.

Such fix can take a week. Especially considering that engineering might be not in a good shape in Skype few years after acquisition from original founders.

Reading the discussion here reminded me of the outage of AT&T's long-distance network back in 1991. After a bit of searching, I found an interesting-looking document at http://faqs.org/rfcs/rfc3439.html entitled "RFC 3439 - Some Internet Architectural Guidelines and Philosophy," which seems relevant here somehow. Though this document outlines considerations pertaining to complexity in Internet backbone architecture, perhaps the overall philosophical questions it poses and guidelines it offers could be instructive as regards the recent Skype meltdown.

As regards the aforementioned 1991 failure of the AT&T long-distance system, which resulted in a service outage of about six hours, the document says:

"The PSTN's SS7 control network provides an interesting example of what can go wrong with a tightly coupled complex system. Outages such as the well publicized 1991 outage of AT&T's SS7 demonstrates the phenomenon: the outage was caused by software bugs in the switches' crash recovery code. In this case, one switch crashed due to a hardware glitch. When this switch came back up, it (plus a reasonably probable timing event) caused its neighbors to crash When the neighboring switches came back up, they caused their neighbors to crash, and so on [NEUMANN] (the root cause turned out to be a misplaced 'break' statement; this is an excellent example of cross- layer coupling). This phenomenon is similar to the phase-locking of weakly coupled oscillators, in which random variations in sequence times plays an important role in system stability [THOMPSON]."

This has absolutely nothing to do with the thundering herd problem. Why are you linking to seemingly random Wikipedia articles?

The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.

True, there's no reason but that doesn't mean it doesn't happen. Many people (I would argue, rightly) equate degraded service to being out of service.

Made-up Scenario: My cluster can handle XXXk users with an SLA of YYms response time. In degraded mode, I'm only handling XXk users with YYYYms response times.

I'm not meeting my SLA for the remaining number of users so I am, in essence, offline.

As to your specific point of "crashing", look at what happened with 37s. Should the server have crashed? No but there was a bug. The reason you add more capacity in the FIRST place is because the existing number of nodes cannot handle the volume. Depending on any number of bugs, issues or configuration your degraded capacity is for all intents and purposes "crashed".

Made-up scenario #2: A single server in your apache configuration can handle 200k concurrent connections reliably with fast response times. Double that load and response times are so long that various devices on the path are timing out the connections as stale. Apache hasn't crashed but it's not really doing anything.

Fast failure is an accepted best practice. Shit, it's baked into Erlang. Kill the process, start a new one and move on. Depending on the nature of the crash, you're doing nothing but churning processes and not actually servicing requests.

The bigger problem is that people don't design for this type of scenario. Static landing pages. Decoupled services instead of monolithic all-in-one containers. Look at github. That's an awesome example of how to degrade service during an outage. Only certain components are "crashed" because everything is fairly decoupled.

Meanwhile there's a guy over here running 4 apps in the same tomcat container that communicate over memory transport with each other or even if he had the common sense to decouple each app, didn't bother to fail fast and was busy spinning up threads trying to communicate with the rest of the services that he can't actually respond to anything externally.

Usually too many requests mean the machine runs out of memory. How it handles that is different, but it usually means the service goes down and might need a restart.