Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skype brief post-mortem

skype.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Skype brief post-mortem

#31

Post by terryjsmith »

dennisgorelik wrote: post #24
barrkel wrote: post #22

Earlier quoted context omitted.

Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?

There is a meaningful distinction between a server that serves 1% of requests and server that is down. I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge. Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node? Shouldn'…

It handling 1% of connections would assume that the network was the bottleneck. You are much more likely to use up the CPU, RAM, or other resources before hitting the maximum number of available sockets. In which case things are swapping or waiting for available CPU time and each individual requests becomes seconds or tens of seconds to get handled. For all intents and purposes, that machine is dead.

As to your second point, I think you are right. I assume that is what the mega-supernode is: a network of machines who's resources are as high as can be to handle all of the connections and try to beat the bottlenecks.

Re: Skype brief post-mortem

#32

Post by tommi »

terryjsmith wrote: post #28
tommi wrote: post #26

Earlier quoted context omitted.

I tend to disagree. Error handling exactly in such cases where you have to bring the network online from a complete halt or other catastrophe is at the very heart of the architecture of these systems. These scenarios should and are at the minds of the architects and coders.

That's a lot easier when it's your network. Skype is obviously designed to be used in "hostile" environments; super nodes are likely expected to go away and come back on a regular basis. Likewise, I don't think it's an unreasonable expectation that there will be a given number of nodes online at any given time. If they all disappear, I doubt there's a contingency plan for that.

From the Skype's status update: "Our engineers are creating new ‘mega-supernodes’ as fast as they can, which should gradually return things to normal."

So they can inject new supernodes in to the system and clients will discover'em gradually. That kind of indicates that they have a mechanism not to DDOS the new mega-supernodes. It's a contingency plan. I fail to see what we are arguing here.

Re: Skype brief post-mortem

#33

Post by terryjsmith »

anonymous246 wrote: post #14

Does anybody know if these supernodes are paid for by Skype or they're simply users' computers being commandeered by Skype (after getting click-through permission). The explanation doesn't ring true to me because AFAIK supernodes are used only if a direct P2P path cannot be established between caller and callee. What we're observing is the inability to sign in and see contacts' status. I always thought that was centr…

This article is a little old now but I imagine the basic technique is still the same:

http://www.h-online.com/security/features/How-Skype-Co-get-r...

As I understand it, for corporate firewalls or routers/firewalls configured to not allow "hole punching", Skype will use the supernode as a relay for all communications.

Re: Skype brief post-mortem

#34

Post by terryjsmith »

tommi wrote: post #32
terryjsmith wrote: post #28

Earlier quoted context omitted.

That's a lot easier when it's your network. Skype is obviously designed to be used in "hostile" environments; super nodes are likely expected to go away and come back on a regular basis. Likewise, I don't think it's an unreasonable expectation that there will be a given number of nodes online at any given time. If they all disappear, I doubt there's a contingency plan for that.

From the Skype's status update: "Our engineers are creating new ‘mega-supernodes’ as fast as they can, which should gradually return things to normal." So they can inject new supernodes in to the system and clients will discover'em gradually. That kind of indicates that they have a mechanism not to DDOS the new mega-supernodes. It's a contingency plan. I fail to see what we are arguing here.

I'm just relaying my experience with P2P network design and with system-wide failures. I agree with you that gradual discovery is possible, but I doubt that's the best case scenario for them depending on how long it takes. They have the worst of both world: if it's slow, their users are going to be extremely frustrated, but if their clients are very quick to discover new nodes, the possibility of them getting swamped is still very real. The status update doesn't talk about their deployment strategy at all so I don't think we can make assumptions; either way has it's own potentially serious issues.

Re: Skype brief post-mortem

#35

Post by dennisgorelik »

terryjsmith wrote: post #17
johndbritton wrote: post #7

"Our engineers are creating new ‘mega-supernodes’ as fast as they can" What a brilliant solution.

Yea, see the above comments. Bringing up individual nodes isn't an option as they would likely be overwhelmed and rate limiting traffic from all of the running instances in a P2P network would be near impossible. Bringing up an entirely new super node network, while insane, is probably their best option right now. Edit: I initially read your post as sarcastic, but I realize now it may not have been. It's definitely o…

HackerNews does not allow deep threads, (http://news.ycombinator.com/item?id=2032770) so I reply here.

Why would supernode that has CPU as a bottleneck stop serving _any_ requests?

Say, CPU is able to serve a thousand requests per second. If supernode receives a million requests per second, then supernode should serve first thousand requests and then ignore the rest, right?

Re: Skype brief post-mortem

#36

Post by terryjsmith »

ddlatham wrote: post #25
terryjsmith wrote: post #2

This seems like a nightmare situation for P2P company. Your super nodes get knocked offline and there's no way to force update them until they come back online, which is sporadic at best (based on my own experience today), which means either manually updating them or completely re-seeding your network (which seems to be the route they're going -- though so far it doesn't appear to be going all that well). As other pe…

There's an important lesson here for P2P system design. As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.

Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond their control.

Re: Skype brief post-mortem

#37

Post by moe »

dennisgorelik wrote: post #24
barrkel wrote: post #22

Earlier quoted context omitted.

Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?

There is a meaningful distinction between a server that serves 1% of requests and server that is down. I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge. Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node? Shouldn'…

I assume that overloaded server still serves some requests.

See http://en.wikipedia.org/wiki/Thundering_herd_problem

Many types of systems need a warm up period before they can realize their full performance. In web applications a controlled warm up is often needed to prime the caches. In P2P applications - which you can't easily "reboot" as a whole - the restoration of a steady state can be much more complex.

Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?

In theory, yes. In practice this seems to be a case of http://en.wikipedia.org/wiki/Cascading_failure

The remaining supernodes either can't handle the aggregate load alone. Or they are being overwhelmed because the re-connection attempts from clients are not evenly distributed.

Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?

In theory, yes. In practice there's probably a lot of http://en.wikipedia.org/wiki/Positive_feedback and perhaps even http://en.wikipedia.org/wiki/Monster_wave going on in the skype network right now.

Re: Skype brief post-mortem

#38

Post by barrkel »

dennisgorelik wrote: post #24
barrkel wrote: post #22

Earlier quoted context omitted.

Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?

There is a meaningful distinction between a server that serves 1% of requests and server that is down. I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge. Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node? Shouldn'…

It usually doesn't work that way. Usually, when your server is overloaded, instead of serving some fraction of the requests in reasonable time, it tries to queue the requests and serve them in order, and as a result response time goes through the roof. Often, the user at the other end of the causal chain starts getting frustrated and refreshes, adding more requests to the queue.

When servers are getting overloaded, you need to throttle the incoming requests aggressively and early, to keep response time reasonable.

Re: Skype brief post-mortem

#39

Post by terryjsmith »

dennisgorelik wrote: post #35
terryjsmith wrote: post #17

Earlier quoted context omitted.

Yea, see the above comments. Bringing up individual nodes isn't an option as they would likely be overwhelmed and rate limiting traffic from all of the running instances in a P2P network would be near impossible. Bringing up an entirely new super node network, while insane, is probably their best option right now. Edit: I initially read your post as sarcastic, but I realize now it may not have been. It's definitely o…

HackerNews does not allow deep threads, ( http://news.ycombinator.com/item?id=2032770 ) so I reply here. Why would supernode that has CPU as a bottleneck stop serving _any_ requests? Say, CPU is able to serve a thousand requests per second. If supernode receives a million requests per second, then supernode should serve first thousand requests and then ignore the rest, right?

Correct, it will stop serving requests (ignore the rest), but the end result is no different (in fact I imagine this is exactly what's happening). That system is dead to the world. According to Wikipedia in Q3 2009 [1] Skype has over 500 million users (Q1 2009 says 42M active at any time); any super node that is processing a thousand requests and then dies (or only processes one request a second) isn't helping anyone.

[1] http://en.wikipedia.org/wiki/Skype#Usage_and_traffic

Re: Skype brief post-mortem

#40

Post by viraptor »

This part is pretty cool, but quite subtle: "Earlier today, we noticed that the number of people online on Skype was falling". I mean that they noticed the number going down, not "people started raising tickets saying it doesn't work".
Post reply on HN