What is amazing here is not that a single network card caused this mess, but that most folks here actually believe this story.
CenturyLink 911 outage was caused by a single network card sending bad packets
131–140 of 166 posts
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#132Earlier quoted context omitted.
From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…
the presence of such a chain is the real root cause, rather than the unhandled sql exception. This is really interesting and something which bugs me about root cause analysis and it's a neat coincidence that this has been quoted relative to an aviation incident. In aviation, incidents and accidents are investigated with the understanding that there is never a single cause of an accident. It's known as the swiss chees…
> like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.
The software industry that programs space craft?
The software industry is vast and not every system involves copious amounts of human decision making. Often the idea of the root system cause, and the root process cause(software construction, operations, etc) cause is separable.
I would say that aviation is almost inverted in the that regard compared to booking systems, banking systems, and most of what software engineers are exposed to. A person can not fly from Dallas to Chicago without many, many human decisions being involved. However, a packet traveling from Dallas to Chicago involves nearly zero new human interactions.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#133Earlier quoted context omitted.
These terms are interchangeable. Although typically one or other is used when speaking about a specific technology. Also see datagram, cell and probably others I’m forgetting right now.
> terms are interchangeable No, use frame when your talking about layer 2, packet when your talking about layer 3, and segment for layer 4. Datagram and protocol data unit (PDU) are general terms that can apply to any layer. A switch forwards frames, while a router routes packets. https://stackoverflow.com/questions/31446777/difference-betw...
You are technically correct. However, the term “Ethernet packet” is commonly used colloquially... it does not seem worth arguing about.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#134>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices" This is of course gibberish. A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. A NIC on their management subnet disrupted their entire network? There are so many levels of absurdity to this. A mangled ethernet frame would be dropped…
> A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable.
OTU also has frames. Optical gear is generally happy to pass along mangled packets :/
> any modern switch from a major vendor has a storm control setting
Look at the switches embedded in optical transport gear. They are pretty rudimentary.
Optical transport gear (L1 networks) are full of impressively clowny behavior.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#135Earlier quoted context omitted.
Who's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it. To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?
We don't, and apparently they didn't either. But it was their job to do so. I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause. We can't know, of course. But I think it would be hard to argue that this is optimal, and that no…
It's hard to work on network problems when the management network is also having problems. That requires dispatching people locally to sites to see what's going on. You're blind in both eyes, not just one.
Look, I'm not defending them or their choices. I'm just saying that this more more nuanced than "They suck they should have known better or spent more money." When you run a network as large as this and with as many "legacy" technologies as they do, things aren't very cut and dry.
NTT probably has the best managed global network with their extensive use of SDN. They run a tight ship and everything goes through their automation frameworks. This requires a huge investment in R&D. Even then, it doesn't cover 100% of their network because "there's that T320 for that customer in Chicago, and the stuff up in Michigan for Dorian's T1." Multiply that times 30 or so to consider size differences between NTT and CenturyLink....and yeah, stuff happens.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#136Earlier quoted context omitted.
YouTube and Netflix cache boxes are important for smaller ISP's. Their cache performance can literally make or break the monthly P&L. That said, I haven't heard about other OTT services offering a caching appliance. And there are more and more OTT streaming services coming. It's great from a consumer perspective, but it's going to be challenging for smaller players.
Solution is to get some datacenters on your network as well. I'm not sure why that didn't happen before.
How would a rural WISP "get some datacenters" on its network? Sorry, I'm not grasping your previous comment.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#137Earlier quoted context omitted.
This is probably the same guy from fuckinglevel3.com. We used level3 for years and sent each other that link almost daily. I guess he had to update after CL bought them :] Don't expect a technical report from CL/L3. We had 60+ mpls/vpls circuits from them and all our reports were very high level. Source: am neteng
I always heard L3 was quite good. Who is if they’re not?
They used to be. It's been a clusterfuck for a few years now with them.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#138Earlier quoted context omitted.
These terms are interchangeable. Although typically one or other is used when speaking about a specific technology. Also see datagram, cell and probably others I’m forgetting right now.
> terms are interchangeable No, use frame when your talking about layer 2, packet when your talking about layer 3, and segment for layer 4. Datagram and protocol data unit (PDU) are general terms that can apply to any layer. A switch forwards frames, while a router routes packets. https://stackoverflow.com/questions/31446777/difference-betw...
I appreciate your insistence on speaking using precise terms. I'm sure I'm not the only one who gauges the knowledge-level and overall attention to detail of a speaker based on their use of precise, correct terminology. It's a useful handshaking technique I use for calibrating the technical content of my speech.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#139>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices" This is of course gibberish. A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. A NIC on their management subnet disrupted their entire network? There are so many levels of absurdity to this. A mangled ethernet frame would be dropped…
> A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. OTU also has frames. Optical gear is generally happy to pass along mangled packets :/ > any modern switch from a major vendor has a storm control setting Look at the switches embedded in optical transport gear. They are pretty rudimentary. Optical transport gear (L1 networks) are full of…
>"OTU also has frames."
Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious.
Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy.
In the RFO Centurylink also mentions - "A decision was made to isolate a device in San Antonio, TX from the network as it seemed to be broadcasting traffic and consuming capacity." Lightwave gear most certainly does have any concept of broadcasts.
Re: CenturyLink 911 outage was caused by a single network card sending bad packets
#140Earlier quoted context omitted.
> A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. OTU also has frames. Optical gear is generally happy to pass along mangled packets :/ > any modern switch from a major vendor has a storm control setting Look at the switches embedded in optical transport gear. They are pretty rudimentary. Optical transport gear (L1 networks) are full of…
>"OTU also has frames." Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious. Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy. In the RFO Centurylink also mentions - "A d…
> Also do you that they would use DWDM for their management network?
Yes, in the form of the Optical Supervisory Channel (OSC), which is built into DWDM gear and generally implemented as Ethernet over SONET.
The OSC can also carry management traffic for other devices (aka datawire).
It's Ethernet, so it has broadcasts...