Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skype brief post-mortem

skype.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Skype brief post-mortem

#41

Post by dennisgorelik »

terryjsmith wrote: post #39
dennisgorelik wrote: post #35

Earlier quoted context omitted.

HackerNews does not allow deep threads, ( http://news.ycombinator.com/item?id=2032770 ) so I reply here. Why would supernode that has CPU as a bottleneck stop serving _any_ requests? Say, CPU is able to serve a thousand requests per second. If supernode receives a million requests per second, then supernode should serve first thousand requests and then ignore the rest, right?

Correct, it will stop serving requests (ignore the rest), but the end result is no different (in fact I imagine this is exactly what's happening). That system is dead to the world. According to Wikipedia in Q3 2009 [1] Skype has over 500 million users (Q1 2009 says 42M active at any time); any super node that is processing a thousand requests and then dies (or only processes one request a second) isn't helping anyone…

If every supernode keeps serving whatever it can (e.g. 1000 requests per second) -- then outage problem would be resolved pretty soon.

But I guess overloaded supernode does not serve anything at all. My guess would be that after 1000 successfully served handshakes clients send 1000 data requests, and most of these data requests are ignored, because supernode is overloaded. End result -- almost no data is distributed.

Most likely the problem is in Skype code for supernodes (almost all Skype clients are supernodes).

I think we should expect Skype hotfix soon.

Re: Skype brief post-mortem

#42

Post by amackera »

terryjsmith wrote: post #36
ddlatham wrote: post #25

Earlier quoted context omitted.

There's an important lesson here for P2P system design. As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.

Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond the…

Do you know of any resources where I could learn about Skype's network structure? I'm very interested in distributed systems, especially ones with the scope of Skype's network.

Re: Skype brief post-mortem

#43

Post by terryjsmith »

amackera wrote: post #42
terryjsmith wrote: post #36

Earlier quoted context omitted.

Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond the…

Do you know of any resources where I could learn about Skype's network structure? I'm very interested in distributed systems, especially ones with the scope of Skype's network.

Sure, so there were a lot of them that we compiled, here are the ones and topics I remember.

Protocol: we used a custom psuedo-TCP protocol built on UDP based on libjingle from Google (used in Google Talk): http://code.google.com/apis/talk/libjingle/file_share.html. Libjingle's filesharing itself was a decent resource for learning some P2P stuff as well.

NATing and routing: we used Skype's UDP hole punching: http://www.h-online.com/security/features/How-Skype-Co-get-r....

Skype configures each install to be a supernode by default and then disables it based on certain criteria (low speeds, behind a NAT/firewall, etc.). Discovery of other network nodes is an entire subject unto itself and a lot of other network discovery protocols are well documented; our system used a central server to track up-time statistically and map clients to one another based on finding a good fit. Happy to answer any questions here or at my e-mail address in profile.

Re: Skype brief post-mortem

#44

Post by christoph »

terryjsmith wrote: post #36
ddlatham wrote: post #25

Earlier quoted context omitted.

There's an important lesson here for P2P system design. As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.

Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond the…

The problem (I believe) is that SkyPE made their own infrastructure so complicated and encrypted that when problems happen it makes it so much harder to deal with. The upside is that they still have the best network around for what they do. When MSN fails like this these days, nobody mentions it, because they're not using MSN, they're using SkyPe.

Re: Skype brief post-mortem

#47

Post by dedward »

dennisgorelik wrote: post #5
terryjsmith wrote: post #2

This seems like a nightmare situation for P2P company. Your super nodes get knocked offline and there's no way to force update them until they come back online, which is sporadic at best (based on my own experience today), which means either manually updating them or completely re-seeding your network (which seems to be the route they're going -- though so far it doesn't appear to be going all that well). As other pe…

Why is it a problem to bring supernode back online?

Supernodes are just guys like you and me with the right amount of bandwidth and the right network connection - if bad code or bad data exposed an existing bug, causing the supernodes in the meshed p2p network to go down - you have a chicken and egg problem.

Skype doens't own the supernodes.... skype works because it uses the user's own resources to help route calls for other users.

Re: Skype brief post-mortem

#48

Post by gojomo »

But why did a "large umber of supernodes" today get "taken offline by a problem affecting some versions of Skype"?

Synchronized bug in that software – some sort of clock overflow, or update from Skype gone awry?

Or a flaw in that software discovered and exploited by others?

There must be a lot more to this story.

Re: Skype brief post-mortem

#49

Post by code_duck »

ddlatham wrote: post #3

Under normal circumstances, there are a large number of supernodes available. Unfortunately, today, many of them were taken offline by a problem affecting some versions of Skype. So, what was the problem affecting versions of Skype?

I doubt that they'd want to tell everyone, since apparently triggering this problem effectively creates a DOS attack against the entire network.

Re: Skype brief post-mortem

#50

Post by brown9-2 »

amackera wrote: post #42
terryjsmith wrote: post #36

Earlier quoted context omitted.

Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond the…

Do you know of any resources where I could learn about Skype's network structure? I'm very interested in distributed systems, especially ones with the scope of Skype's network.

Found this paper posted on reddit for this topic:

"An Experimental Study of the Skype Peer-to-Peer VoIP System" http://saikat.guha.cc/pub/iptps06-skype/

Post reply on HN