Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

Post reply on HN
Print view View original

91–100 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#91

Post by bogomipz »

>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices"

This is of course gibberish. A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable.

A NIC on their management subnet disrupted their entire network? There are so many levels of absurdity to this.

A mangled ethernet frame would be dropped if the CRC was incorrect. A "show int" on a switch would have shown drop counters incrementing. If it was a broadcast storm it also should have been obvious which device was sending an outsized amount of traffic to the all 1's address. Management networks are generally low traffic - ssh and some SNMP. It would have should have been obvious looking at interface graphs by TX on the management network.

Further any modern switch from a major vendor has a storm control setting which disables a port when it goes beyond a certain threshold for either broadcast, multicast or unicast. Even if storm control wasn't enabled it would have been trivial to do so, find the offending port and work backwards from there.

>"A polling filter was applied to adjust the way packets were received in the network equipment"

"Polling filter" is not even an idiomatic network engineering term. I'll assume this means an access list. So it took them 50 hours to apply an ACL? And this required engaging the hardware vendor?

This is a garbage RFO even if its not meant for a technical audience. It sounds like the real RFO is due to incompetence, bad network design and probably a horrid corporate culture shaped by fear, silos and CYA at this company.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#92

Post by loeg »

pooppaint wrote: post #65
loeg wrote: post #48

Earlier quoted context omitted.

Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical…

You sure it’s plaintext? GPON networks should be using AES128 to each ONT.

No, not sure. I wasn't aware of that, thanks. Do you know more about the key negotiation and encryption protocol? Wikipedia doesn't have even mention AES. Thanks.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#93

Post by bjt »

JumpCrisscross wrote: post #85
noir_lord wrote: post #64

Earlier quoted context omitted.

Seems strange how bad ISP's are in the US compared to the UK. I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

> Seems strange how bad ISP's are in the US compared to the UK UK average fixed-line download speeds in Q4 2017 were about 26 Mbps [1]. The same Q2/Q3 2018 statistic for the United States was over 96 [2]. American broadband is crappy. But in mean technological leadership, it’s ahead of the UK. (At the leading edge, I get 400/35 for $80 in Manhattan.) [1] http://www.speedtest.net/reports/united-kingdom/#fixed [2] http…

I wish they'd do the median rather than the mean. Having a small percentage of people with super high speeds will drag the mean up, but it doesn't change the experience for the typical consumer.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#94

Post by mikeash »

jka wrote: post #46
mikeash wrote: post #2

They’re now filtering bad packets so this can’t happen again. No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days .

Worth giving some credit though - mitigation is important for something as critical as 911 service. Hopefully they're also tracking the design flaws, and yes that's worth following (and asking whether they're planning to do so?), but bear in mind people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future).

I completely agree. I wouldn’t expect them to have this fix already, since it probably requires some major work. But some mention that they’ll be looking into it would certainly be nice.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#95

Post by sangnoir »

jonathankoren wrote: post #72
noir_lord wrote: post #64

Earlier quoted context omitted.

Seems strange how bad ISP's are in the US compared to the UK. I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

What's really strange is how the crappy broadband is continuously justified in the US.

The belief in American exceptionalism isn't limited to just positives.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#96

Post by ams6110 »

X-Istence wrote: post #75
sethdj wrote: post #40

Earlier quoted context omitted.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

I live alone, and generally I do about ~400GB of data transfer on Comcast a month. I work from home, and spend a lot of time downloading Docker containers and other such things to do my work. Last month apparently I hit a new high for me: 890GB. Lots of devices needing updates/games downloaded... I am really surprised that with a 7 person household you don't go through your data even faster. I even run a local cachin…

Any option to host your docker stuff on a VPS or other system that isn't using your home internet connection?

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#97

Post by CydeWeys »

JumpCrisscross wrote: post #85
noir_lord wrote: post #64

Earlier quoted context omitted.

Seems strange how bad ISP's are in the US compared to the UK. I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

> Seems strange how bad ISP's are in the US compared to the UK UK average fixed-line download speeds in Q4 2017 were about 26 Mbps [1]. The same Q2/Q3 2018 statistic for the United States was over 96 [2]. American broadband is crappy. But in mean technological leadership, it’s ahead of the UK. (At the leading edge, I get 400/35 for $80 in Manhattan.) [1] http://www.speedtest.net/reports/united-kingdom/#fixed [2] http…

I'm in Manhattan too, and with FIOS my options are 100/100 for $40, 300/300 for $60, or 940/880 for $80. Admittedly those prices might be first year discounted prices.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#98

Post by CydeWeys »

jandrese wrote: post #31
CydeWeys wrote: post #30

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

I'd say the majority of consumers stick with their existing suppliers out of sheer inertia. It takes time, effort, and hassle to switch, and absent any meaningful change in status quo (like a prolonged outage or sharp price increase), most people will coast. Especially when they're not technically inclined, as the difference in service will be less meaningful or apparent to them.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#99

Post by throw0u1t »

Jach wrote: post #61
carlsborg wrote: post #8

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…

I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#100

Post by mey »

Jach wrote: post #61
carlsborg wrote: post #8

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…

My favorite is constructors of things that taken input/output streams that themselves can throw. It becomes very verbose to ensure that the unwinding happens correctly.
Post reply on HN