DE-Cix – Power failure leads to Downtime in germany
heimatzeitung.de
DE-Cix – Power failure leads to Downtime in germany
1–10 of 12 posts
Re: DE-Cix – Power failure leads to Downtime in germany
#2Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down.
I am concerned that this even happened in a place important like this.
pic related: http://img.pr0gramm.com/2018/04/09/9286baa4b2261184.jpg
The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two hours (speculation).
Shouldn't we decentralize more?
Re: DE-Cix – Power failure leads to Downtime in germany
#3Kinda fun to see the increase in AMS-IX traffic: https://ams-ix.net/technical/statistics
Keep in mind this is only one of the many DE-CIX datacenters, they exist in at least 8 datacenters in Frankfurt: https://twitter.com/goebelmeier/status/983453047543214081
Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage.
Re: DE-Cix – Power failure leads to Downtime in germany
#4Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down. I am concerned that this even happened in a place important like this. pic related: http://img.pr0gramm.com/2018/04/09/9286baa4b2261184.jpg The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two…
Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage. The whole of DE-CIX isn't down.
Re: DE-Cix – Power failure leads to Downtime in germany
#5Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down. I am concerned that this even happened in a place important like this. pic related: http://img.pr0gramm.com/2018/04/09/9286baa4b2261184.jpg The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two…
> The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two hours (speculation).
More likely, the timestamp in the picture is UTC, which corresponds to 22:00 CEST, cf. [0]
Re: DE-Cix – Power failure leads to Downtime in germany
#6We were hit by this.
In fact, we're hit by this again, down again at 22:09:19 EST for me and still down. Their website is broken too (https://www.de-cix.net/) and their email servers are offline which makes contacting them amusing.
Not much to be done, really, fortunately the impact for us is very trivial.
Re: DE-Cix – Power failure leads to Downtime in germany
#7Kinda fun to see the increase in AMS-IX traffic: https://ams-ix.net/technical/statistics Keep in mind this is only one of the many DE-CIX datacenters, they exist in at least 8 datacenters in Frankfurt: https://twitter.com/goebelmeier/status/983453047543214081 Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage.
In fairness, we're in FRA8 but DE-CIX is not directly, so we're connected to their switches in FRA5.
Re: DE-Cix – Power failure leads to Downtime in germany
#8Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down. I am concerned that this even happened in a place important like this. pic related: http://img.pr0gramm.com/2018/04/09/9286baa4b2261184.jpg The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two…
Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage. The whole of DE-CIX isn't down.
> Shouldn't we decentralize more?
> The whole of DE-CIX isn't down.
Largely agreed with you, though as general advice I would say that redundancy by adding more exchanges to your network serves you better than adding diverse connections to a single exchange.
Re: DE-Cix – Power failure leads to Downtime in germany
#9Just received an email update from INTERXION acknowledging they did indeed lose power again at FRA5. This leaves us in a bit of a mess.
Re: DE-Cix – Power failure leads to Downtime in germany
#10Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down. I am concerned that this even happened in a place important like this. pic related: http://img.pr0gramm.com/2018/04/09/9286baa4b2261184.jpg The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two…
Well, now we know that significant portion of that facility are 1+0 for power systems. In a true 1+1 setup with air gap between A and B side power systems this wouldn't have happened.