This is infrastructure on which lives depend. Who exactly is liable here outside of a FCC fine? Its just insane how software errors are apparently more or less considered equivalent to "higher power" losses. This all not mentioning the lack of a backup plan for a 50h outage.
CenturyLink 911 outage was caused by a single network card sending bad packets
121–130 of 166 posts
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#122I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…
> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debtCost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this outage on cost cutting.
You're simply bashing a large company that had a bad day. They might deserve your vitriol for plenty of other reasons in other cases, but you can't blame this outage on cost cutting.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#123Earlier quoted context omitted.
> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running: Especially now with cordcutting becoming the norm and traditional cable subscription number dropping, large providers are scrambling to find ways to extract profit. With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are act…
> With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access. Don’t most ISPs cache this sort of content closer to their customers? Netflix at least offers to install hardware that does this.
YouTube and Netflix cache boxes are important for smaller ISP's. Their cache performance can literally make or break the monthly P&L. That said, I haven't heard about other OTT services offering a caching appliance. And there are more and more OTT streaming services coming. It's great from a consumer perspective, but it's going to be challenging for smaller players.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#124I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…
> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt Cost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this ou…
It seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as that seems like a plausible thing an IT/networking department would've gotten around to with more funding, and more and higher-skilled, higher-paid veteran staff who could pre-emptively identify and fix this issue through experience and testing.
Any time something really bad happens that could have been prevented, it's reasonable to ask why it wasn't prevented. This wasn't some black swan event, just a routine hardware failure that they didn't have defenses against.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#125Earlier quoted context omitted.
Education and healthcare are of high quality in the US, generally speaking. They are just super expensive.
The Radio Eriwan joke writes itself.
Oh man, those still hold up too. “In principle yes, but unfortunately nobody here has the education to be sure.”
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#126Earlier quoted context omitted.
> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt Cost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this ou…
It seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as t…
Who's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it.
To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#127Earlier quoted context omitted.
> With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access. Don’t most ISPs cache this sort of content closer to their customers? Netflix at least offers to install hardware that does this.
YouTube and Netflix cache boxes are important for smaller ISP's. Their cache performance can literally make or break the monthly P&L. That said, I haven't heard about other OTT services offering a caching appliance. And there are more and more OTT streaming services coming. It's great from a consumer perspective, but it's going to be challenging for smaller players.
Solution is to get some datacenters on your network as well. I'm not sure why that didn't happen before.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#128Earlier quoted context omitted.
Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.
Seems strange how bad ISP's are in the US compared to the UK. I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).
Depends on where you are. I’m paying $80/month for 940/940 fiber with no cap or throttling. A few miles down the road the only option is slow and unreliable cable through Comcast.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#129Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure. A single interface shutdown command would have been all that was needed if they had remote access.
> Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure.
Or that said infrastructure may exist, but not redundant... I mean, RS232-over-IP or RS232-over-ISDN boxes are no secret sauce, but when their access line is routed over the same thing the box is supposed to remote-manage, then one has problems.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#130Earlier quoted context omitted.
It seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as t…
Who's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it. To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?
We don't, and apparently they didn't either. But it was their job to do so.
I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause.
We can't know, of course. But I think it would be hard to argue that this is optimal, and that nobody at CenturyLink ever suggested it could be better if they spent a bit more money.