Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

11–20 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#11

Wonder how they found it... I bet that was some unhappy hours looking for it.

That's the "I found a 15-year-old bug in our NIC" story that I'd love to read but which will probably never be told. And whoever designed the system is busy obfuscating that on their resume this morning.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#12

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

Yep. It sounds like the packets were coming from this nic and the backbone routers had no logic to filter out those kinds of invalid packets. No SRC/DST/TTL should be a big red flag, but it seems like they didn't have a rule for that.

Talk about a needle in a hay stack.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#13

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

It's affect, not effect.

https://www.grammarly.com/blog/affect-vs-effect/

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#14

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

I used to look after an office location that had three racks with older generation HP Procurve switches. About once every 4-6 months, the whole lot would go haywire and the only way to settle down the network was to switch them all off, wait, and then power up again. We never did track down the root cause and could never make it happen on demand,

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#15

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

One example might be "ip helper" setups in a Cisco router. People use it to forward DHCP requests over a WAN. One broken SIP device, no flood control, and...bam, your WAN is hosed.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#17

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

It will be almost impossible to get a good idea of the actual RCA from public reports, since even if they are accurate, we would need a complete understanding of the topology of the network in question to understand them. That said, the public descriptions will be filtered through so many layers of PR and lawyers as to be incomprehensible. "invalid frame packet"? Were they invalid frames or invalid packets? As fun as it is to gawk, I don't think we'll ever get a good picture of what happened and what CenturyLink did wrong, if anything, to allow such an outage to take place.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#18

Wonder how they found it... I bet that was some unhappy hours looking for it.

Once they knew what they were looking for it was probably pretty quick to find. A network card that's just vomiting all over the network is loud and visible by definition.
Post reply on HN