CenturyLink 911 outage was caused by a single network card sending bad packets

They’re now filtering bad packets so this can’t happen again.

No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days.

Someone was displeased - https://fuckingcenturylink.com/

I'd love to see an in-depth technical analysis of the outage.

The descriptions so far about this problem are either at 30000 foot vagueness or they're in technical shorthand that just assumes the audience is 100% pro network engineers.

Don't "bad packets" get dropped at the first switch? Isn't that one of the main benefits of packet based switching?

Was this even an ethernet packet or something else like an optical transport protocol (eg OTN)?

How does a single network card emitting bad packets effect other sites?

> investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card.

> Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes.

And then

> By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service.

And finally

> The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.

Looks like the root cause analysis has a way to go. Addendum says:

> The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner. The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment. The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.