Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skype brief post-mortem

skype.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Skype brief post-mortem

#61

Post by JoachimSchipper »

barrkel wrote: post #38
dennisgorelik wrote: post #24

Earlier quoted context omitted.

There is a meaningful distinction between a server that serves 1% of requests and server that is down. I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge. Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node? Shouldn'…

It usually doesn't work that way. Usually, when your server is overloaded, instead of serving some fraction of the requests in reasonable time, it tries to queue the requests and serve them in order, and as a result response time goes through the roof. Often, the user at the other end of the causal chain starts getting frustrated and refreshes, adding more requests to the queue. When servers are getting overloaded, y…

In network engineering, systems quite often use RED (http://en.wikipedia.org/wiki/Random_early_detection) - proactively drop a percentage of connections/packets/... when loads begins to climb, scaling from 0% (any easily handled load) to 100% (at capacity). In practice, this results in pretty robust systems. (But note that TCP et al. have mechanisms that help this, e.g. slowing down when packets are lost and retrying a connection with a backoff.)

Re: Skype brief post-mortem

#62

Post by jemfinch »

lusis wrote: post #57
jemfinch wrote: post #55

Earlier quoted context omitted.

This has absolutely nothing to do with the thundering herd problem. Why are you linking to seemingly random Wikipedia articles? The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.

True, there's no reason but that doesn't mean it doesn't happen. Many people (I would argue, rightly) equate degraded service to being out of service. Made-up Scenario: My cluster can handle XXXk users with an SLA of YYms response time. In degraded mode, I'm only handling XXk users with YYYYms response times. I'm not meeting my SLA for the remaining number of users so I am, in essence, offline. As to your specific po…

You're missing a fundamental difference between degraded service and being out of service: if an overloaded server handles its maximum number of clients but doesn't crash, then you have time to bring up additional servers to share the load. If an overloaded server crashes instead of limiting the number of clients it serves, you have to staunch the flow of clients further upstream before you can bring more servers up to share the load. This is what the GGGP is saying, and it seems like everyone (including you) are missing his point.

Degrading service, or more accurately, serving the maximum number of clients you can serve and no more is absolutely not the same thing as being down.

Failing/crashing rather than limiting the number of clients you serve is absolutely not established best practice. It's poor practice, in fact.

Re: Skype brief post-mortem

#63

Post by jemfinch »

moe wrote: post #60
jemfinch wrote: post #55

Earlier quoted context omitted.

This has absolutely nothing to do with the thundering herd problem. Why are you linking to seemingly random Wikipedia articles? The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.

This has absolutely nothing to do with the thundering herd problem. The skype outage probably not. This was in response to the thread going on about overloaded webapps - thundering herd is a common issue there. The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service. Nobody used the word "crash" but graceful degradation is quite hard. You have…

Thundering herd is an efficiency problem on certain implementations of UNIX servers. It's not a common reason why servers crash when overloaded.

Graceful degradation is not hard. Serve N clients, and no more. It's an extra conditional in the event loop. Am I serving N or more clients right now? Then don't accept new connections (or accept(2) them and immediately close them to avoid a backup in the listen queue).

Re: Skype brief post-mortem

#64

Post by AgentConundrum »

igravious wrote: post #58
AgentConundrum wrote: post #23

Earlier quoted context omitted.

That's what I've been wondering all day. I didn't have a problem with Skype being down; I had a problem with Skype having exploded this morning, screaming about uncaught exceptions as it did. Your quote was an "aha" moment for me.

So did I, were you travelling at the time? I was in an airport and I figured that it is the weird you have an internet connection but it only takes you to a pay per usage landing page and no other page kind of problem that made Skype barf. I happened to a Java network client too at the same time. I can send you logs Skype!

Nope, not traveling. I was just sitting at home and it exploded.

Re: Skype brief post-mortem

#65

Post by moe »

jemfinch wrote: post #63
moe wrote: post #60

Earlier quoted context omitted.

This has absolutely nothing to do with the thundering herd problem. The skype outage probably not. This was in response to the thread going on about overloaded webapps - thundering herd is a common issue there. The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service. Nobody used the word "crash" but graceful degradation is quite hard. You have…

Thundering herd is an efficiency problem on certain implementations of UNIX servers. It's not a common reason why servers crash when overloaded. Graceful degradation is not hard. Serve N clients, and no more. It's an extra conditional in the event loop. Am I serving N or more clients right now? Then don't accept new connections (or accept(2) them and immediately close them to avoid a backup in the listen queue).

Thundering herd is an efficiency problem on certain implementations of UNIX servers.

The accept()-issue you're referring to is merely a popular manifestation of the thundering herd problem.

The term is nowadays also used in a broader sense to describe similar situations in distributed systems.

It's not a common reason why servers crash when overloaded.

Again, nobody is talking about servers crashing here.

We are talking about web applications (which usually consist of multiple servers talking to each other) degrading to the point of not serving a meaningful rate of requests anymore.

In that context the role of the "thundering herd" is often taken by your own users, who will furiously smash their reload-buttons to get past those 500-errors.

It's an extra conditional in the event loop. Am I serving N or more clients right now?

You're talking about implementation details of a socket server. I was talking about entire applications.

Re: Skype brief post-mortem

#66

Post by lusis »

jemfinch wrote: post #62
lusis wrote: post #57

Earlier quoted context omitted.

True, there's no reason but that doesn't mean it doesn't happen. Many people (I would argue, rightly) equate degraded service to being out of service. Made-up Scenario: My cluster can handle XXXk users with an SLA of YYms response time. In degraded mode, I'm only handling XXk users with YYYYms response times. I'm not meeting my SLA for the remaining number of users so I am, in essence, offline. As to your specific po…

You're missing a fundamental difference between degraded service and being out of service: if an overloaded server handles its maximum number of clients but doesn't crash, then you have time to bring up additional servers to share the load. If an overloaded server crashes instead of limiting the number of clients it serves, you have to staunch the flow of clients further upstream before you can bring more servers up…

I'm approaching it from the business perspective. If sipport staff are getting calls for poor performance then the product isn't working. End of story. Business units don't care about shades of gray and degraded service. It's binary for them. It's either working or it isn't. Couple that with financial penalties for latency and degraded service means even less. Either you're meeting your SLA or you aren't.
Post reply on HN