Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

Post reply on HN
Print view View original

21–30 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#21

Post by bch »

carlsborg wrote: post #8

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#23

Post by sitkack »

linker3000 wrote: post #14
dec0dedab0de wrote: post #9

Earlier quoted context omitted.

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

I used to look after an office location that had three racks with older generation HP Procurve switches. About once every 4-6 months, the whole lot would go haywire and the only way to settle down the network was to switch them all off, wait, and then power up again. We never did track down the root cause and could never make it happen on demand,

Could be a totally valid solution ... for an equipment rental place. Pretty sure people died that didn't have to with this 911 outage.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#24

Post by solarengineer »

I had faced something like this a few years ago: https://dynamicproxy.livejournal.com/46862.html

Summary: the specific Symantec disk imaging software was partly loaded via PXE boot, and that machine started to flood the network with bad packets. Switching the computer off for a few seconds didn't help, since the SMPS capacitors still held current - enough to keep the card alive and for the sysadmins to not suspect that computer!

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#25

Post by azinman2 »

nubb wrote: post #7
TheWoodsy wrote: post #4

Someone was displeased - https://fuckingcenturylink.com/ I'd love to see an in-depth technical analysis of the outage.

This is probably the same guy from fuckinglevel3.com. We used level3 for years and sent each other that link almost daily. I guess he had to update after CL bought them :] Don't expect a technical report from CL/L3. We had 60+ mpls/vpls circuits from them and all our reports were very high level. Source: am neteng

I always heard L3 was quite good. Who is if they’re not?

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#26

Post by scruffyherder »

nubb wrote: post #7
TheWoodsy wrote: post #4

Someone was displeased - https://fuckingcenturylink.com/ I'd love to see an in-depth technical analysis of the outage.

This is probably the same guy from fuckinglevel3.com. We used level3 for years and sent each other that link almost daily. I guess he had to update after CL bought them :] Don't expect a technical report from CL/L3. We had 60+ mpls/vpls circuits from them and all our reports were very high level. Source: am neteng

I have a uh .. 'client' that uses CL/L3 and I swear they have outages WEEKLY. Like they should give notice when we can use the circuit vs when it's going down. again.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#27

Post by extrapickles »

crispyambulance wrote: post #6

The descriptions so far about this problem are either at 30000 foot vagueness or they're in technical shorthand that just assumes the audience is 100% pro network engineers. Don't "bad packets" get dropped at the first switch? Isn't that one of the main benefits of packet based switching? Was this even an ethernet packet or something else like an optical transport protocol (eg OTN)?

They generally only get dropped if they have an invalid checksum.

Since checksums are hardware accelerated, the invalid packet probably had a valid checksum applied to it.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#28

Post by extrapickles »

CamTin wrote: post #17
carlsborg wrote: post #8

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

It will be almost impossible to get a good idea of the actual RCA from public reports, since even if they are accurate, we would need a complete understanding of the topology of the network in question to understand them. That said, the public descriptions will be filtered through so many layers of PR and lawyers as to be incomprehensible. "invalid frame packet"? Were they invalid frames or invalid packets? As fun as…

My reading of the tea leaves is that the invalid packets where for some low level control protocol.

I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad packets, so after a delay it would enable the trunk again and get the invalid packets again.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#29

Post by ecp9 »

Someone1234 wrote: post #5

Makes you wonder how secure their backhaul really is? If the whole thing is a single flat logical network (one that could allow bad packets to propagate as we witnessed) that would suggest it is also quite vulnerable to malicious actions. It is all well and good applying a filter, but that seems like a bandaid fix. Why is equipment even able to talk that has no reason to do so? Seems like they've put convenience over…

Much networking equipment is not designed to handle malicious or bizarre traffic. TCP/IP is amazingly brittle, and often fails on me in surprising ways that the standards say should never ever happen.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#30

Post by CydeWeys »

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.)

This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink, as well as my SO giving them a Roku for the holidays and setting them up with streaming services, which they got lots of practice using in the two whole days the cable TV didn't work.

Guess who's cutting the cord and switching to Internet-only service from Verizon FIOS at a much cheaper rate now?

This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running: You get huge costly outages revealing your basic incompetence and customers fleeing to superior competitors forever.

Post reply on HN