Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

Post reply on HN
Print view View original

31–40 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#31

Post by jandrese »

CydeWeys wrote: post #30

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#32

Post by JustSomeNobody »

mikeash wrote: post #2

They’re now filtering bad packets so this can’t happen again. No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days .

> ... can’t happen again

Gotta love this.

> No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days.

Yep, they put a bandage on it and called it good.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#33

Post by jandrese »

extrapickles wrote: post #28
CamTin wrote: post #17

Earlier quoted context omitted.

It will be almost impossible to get a good idea of the actual RCA from public reports, since even if they are accurate, we would need a complete understanding of the topology of the network in question to understand them. That said, the public descriptions will be filtered through so many layers of PR and lawyers as to be incomprehensible. "invalid frame packet"? Were they invalid frames or invalid packets? As fun as…

My reading of the tea leaves is that the invalid packets where for some low level control protocol. I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad pack…

I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes...

I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on big networks.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#34

Post by jandrese »

lykr0n wrote: post #12
dec0dedab0de wrote: post #9

Earlier quoted context omitted.

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

Yep. It sounds like the packets were coming from this nic and the backbone routers had no logic to filter out those kinds of invalid packets. No SRC/DST/TTL should be a big red flag, but it seems like they didn't have a rule for that. Talk about a needle in a hay stack.

Traditional IP filters are typically useless against packet storms because all of the packets are technically valid, there are just way too many of them.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#35

Post by SlowRobotAhead »

mikeash wrote: post #2

They’re now filtering bad packets so this can’t happen again. No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days .

>a single piece of malfunctioning hardware

Just wait until writes Bloomberg that this was because China had put a secret microchip on a network card. Months of entertainment to follow.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#36

Post by waynesonfire »

jandrese wrote: post #31
CydeWeys wrote: post #30

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

centurylink fiber is pretty great as well

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#37

Post by [deleted] »

waynesonfire wrote: post #36
jandrese wrote: post #31

Earlier quoted context omitted.

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

centurylink fiber is pretty great as well

[deleted]

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#38

Post by kijiki »

extrapickles wrote: post #27
crispyambulance wrote: post #6

The descriptions so far about this problem are either at 30000 foot vagueness or they're in technical shorthand that just assumes the audience is 100% pro network engineers. Don't "bad packets" get dropped at the first switch? Isn't that one of the main benefits of packet based switching? Was this even an ethernet packet or something else like an optical transport protocol (eg OTN)?

They generally only get dropped if they have an invalid checksum. Since checksums are hardware accelerated, the invalid packet probably had a valid checksum applied to it.

If cut-through is enabled in the switch, it won't even drop on bad checksum, since by the time the switch can tell the checksum is wrong, it has already forwarded the entire packet.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#39

Post by Trisell »

jandrese wrote: post #31
CydeWeys wrote: post #30

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

At least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#40

Post by sethdj »

Trisell wrote: post #39
jandrese wrote: post #31

Earlier quoted context omitted.

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

At least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.
Post reply on HN