Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skype brief post-mortem

skype.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Skype brief post-mortem

#21

Post by terryjsmith »

tommi wrote: post #18
terryjsmith wrote: post #15

Earlier quoted context omitted.

Since it seems a good chunk of the super nodes went down, I imagine every running unconnected Skype instance is checking all of the super nodes it can find (or actively searching for them). Changing the logic at this point isn't really an option for them.

Well, even if we were to talk about this point, it actually might be. I have no knowledge of Skype architecture, so I'm totally guessing here. But, those unconnected Skype instances have to have some kind of directory of supernodes: either dynamic or static. If it's dynamic, that is from Skype's servers, it can be affected.

That's fair, but even if it is dynamic, I doubt it's checked on a regular basis; I doubt it's a scenario they've even considered before now. This would be on par with the DNS root servers changing and somehow letting everyone know about it on the fly.

Re: Skype brief post-mortem

#22

Post by barrkel »

dennisgorelik wrote: post #19
terryjsmith wrote: post #13

Earlier quoted context omitted.

When this happens to our websites (when all servers go down), we need to rate limit and/or shut down traffic at the load balancer level as we bring things back online, otherwise everything just continues to get swamped and goes right back down. This would be nearly impossible in a P2P network and coordinating it between locations would be an even bigger nightmare. I imagine this is why turning on an entirely new netw…

Why would overloaded server go down? Shouldn't it simply stop serving incoming requests if it's overloaded?

Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?

Re: Skype brief post-mortem

#23

Post by AgentConundrum »

ddlatham wrote: post #3

Under normal circumstances, there are a large number of supernodes available. Unfortunately, today, many of them were taken offline by a problem affecting some versions of Skype. So, what was the problem affecting versions of Skype?

That's what I've been wondering all day. I didn't have a problem with Skype being down; I had a problem with Skype having exploded this morning, screaming about uncaught exceptions as it did.

Your quote was an "aha" moment for me.

Re: Skype brief post-mortem

#24

Post by dennisgorelik »

barrkel wrote: post #22
dennisgorelik wrote: post #19

Earlier quoted context omitted.

Why would overloaded server go down? Shouldn't it simply stop serving incoming requests if it's overloaded?

Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?

There is a meaningful distinction between a server that serves 1% of requests and server that is down.

I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge.

Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node?

Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?

Re: Skype brief post-mortem

#25

Post by ddlatham »

terryjsmith wrote: post #2

This seems like a nightmare situation for P2P company. Your super nodes get knocked offline and there's no way to force update them until they come back online, which is sporadic at best (based on my own experience today), which means either manually updating them or completely re-seeding your network (which seems to be the route they're going -- though so far it doesn't appear to be going all that well). As other pe…

There's an important lesson here for P2P system design.

As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.

Re: Skype brief post-mortem

#26

Post by tommi »

terryjsmith wrote: post #21
tommi wrote: post #18

Earlier quoted context omitted.

Well, even if we were to talk about this point, it actually might be. I have no knowledge of Skype architecture, so I'm totally guessing here. But, those unconnected Skype instances have to have some kind of directory of supernodes: either dynamic or static. If it's dynamic, that is from Skype's servers, it can be affected.

That's fair, but even if it is dynamic, I doubt it's checked on a regular basis; I doubt it's a scenario they've even considered before now. This would be on par with the DNS root servers changing and somehow letting everyone know about it on the fly.

I tend to disagree. Error handling exactly in such cases where you have to bring the network online from a complete halt or other catastrophe is at the very heart of the architecture of these systems. These scenarios should and are at the minds of the architects and coders.

Re: Skype brief post-mortem

#27

Post by switch007 »

zinssmeister wrote: post #4

does anyone know why skype picked German as the only translation of this message?

My awful Spanish translation:

A principios de hoy, nos dimos cuenta de que el número de usarios conectado a Skype empezó caer. Esto no es normal ni esperado así que nos pusimos a investigarlo.

Skype no es como un red de telefonía convencional ni red de mensajería instantánea - en cambio, para que todo vaya bien depende de millones de conexiones entre los computadores y los teléfonos. Unos de esas computadores se llaman "supernodes" - funcionan como una agenda de direcciones por Skype. Si quieres hablar con alguien y Skype no lo puede localizar inmediatamente, tu computador o tu móvil intentará localizar un "supernode" para descubrir la manera de contactarle.

Normalmente hay un montón de supernodes disponibles. Hoy, por desgracia, muchos no eran conectados debido a una problema que afectaba algunos versiones de Skype.

¿Qué estamos haciendo para mejorar la situación? Nuestros ingenieros están haciendo nuevos "mega-supernodes" lo cuanto antes, que con un poco de suerte va a hacer que todo se normalice.

Re: Skype brief post-mortem

#28

Post by terryjsmith »

tommi wrote: post #26
terryjsmith wrote: post #21

Earlier quoted context omitted.

That's fair, but even if it is dynamic, I doubt it's checked on a regular basis; I doubt it's a scenario they've even considered before now. This would be on par with the DNS root servers changing and somehow letting everyone know about it on the fly.

I tend to disagree. Error handling exactly in such cases where you have to bring the network online from a complete halt or other catastrophe is at the very heart of the architecture of these systems. These scenarios should and are at the minds of the architects and coders.

That's a lot easier when it's your network. Skype is obviously designed to be used in "hostile" environments; super nodes are likely expected to go away and come back on a regular basis. Likewise, I don't think it's an unreasonable expectation that there will be a given number of nodes online at any given time.

If they all disappear, I doubt there's a contingency plan for that.

Re: Skype brief post-mortem

#29

Post by switch007 »

switch007 wrote: post #27
zinssmeister wrote: post #4

does anyone know why skype picked German as the only translation of this message?

My awful Spanish translation: A principios de hoy, nos dimos cuenta de que el número de usarios conectado a Skype empezó caer. Esto no es normal ni esperado así que nos pusimos a investigarlo. Skype no es como un red de telefonía convencional ni red de mensajería instantánea - en cambio, para que todo vaya bien depende de millones de conexiones entre los computadores y los teléfonos. Unos de esas computadores se llam…

Google Translate does a pretty awesome job too, fwiw.

Re: Skype brief post-mortem

#30

Post by oiuytriuytr »

anonymous246 wrote: post #14

Does anybody know if these supernodes are paid for by Skype or they're simply users' computers being commandeered by Skype (after getting click-through permission). The explanation doesn't ring true to me because AFAIK supernodes are used only if a direct P2P path cannot be established between caller and callee. What we're observing is the inability to sign in and see contacts' status. I always thought that was centr…

In theory they can be regular users computers - but unless you have the only PC connected to some small outpost on the internet it's unlikely that it would be you.

Generally it's either a skype or ISP owned machine

Post reply on HN