Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

Post reply on HN
Print view View original

151–160 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#151

Post by vmchale »

jka wrote: post #46
mikeash wrote: post #2

They’re now filtering bad packets so this can’t happen again. No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days .

Worth giving some credit though - mitigation is important for something as critical as 911 service. Hopefully they're also tracking the design flaws, and yes that's worth following (and asking whether they're planning to do so?), but bear in mind people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future).

> people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future).

yes, clearly knocking out 911 service for millions of people isn't a problem. won't someone think of the poor programmers??

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#152

Post by jandrese »

loeg wrote: post #87
ransford wrote: post #79

Earlier quoted context omitted.

I've been getting perfectly adequate IPv6 via their 6rd gateway (on fiber, in Seattle). I wrote some nerd notes on using one's own wifi router and setting up IPv6: http://b.tra.in/2015/07/notes-on-centurylink-fiber.html

6rd is pretty awful, generally, and I couldn't get Centurylink's to work last time I tried. I appreciate the notes and will take a look, thank you. (One big problem I've experienced in the past is that any time you configure both IPv6 and IPv4, all programs prefer IPv6. But tunneled IPv6 has much worse performance than the native IPv4, so you really want applications to prefer IPv4 in a tunneled v6 environment.)

I still use Hurricane Electric tunnels fir my v6 and ping times/throughout are so close as to be in the noise for most everything. It probably helps that I'm using a tunnel endpoint literally 10 miles from my house however.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#153

Post by packet_nerd »

icedchai wrote: post #133
packet_nerd wrote: post #84

Earlier quoted context omitted.

> terms are interchangeable No, use frame when your talking about layer 2, packet when your talking about layer 3, and segment for layer 4. Datagram and protocol data unit (PDU) are general terms that can apply to any layer. A switch forwards frames, while a router routes packets. https://stackoverflow.com/questions/31446777/difference-betw...

You are technically correct. However, the term “Ethernet packet” is commonly used colloquially... it does not seem worth arguing about.

Yeah, but the point is you wouldn't expect a competent network engineer to use the words "packet frames". And incompetence would seem to fit with the facts of the case, i.e. 911 was down for 2 days.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#154

Post by ecp9 »

drcross wrote: post #104
ecp9 wrote: post #29

Earlier quoted context omitted.

Much networking equipment is not designed to handle malicious or bizarre traffic. TCP/IP is amazingly brittle, and often fails on me in surprising ways that the standards say should never ever happen.

I don't think that's fair to say. Billions of people unlock their phone or log in to their computers every morning and everything works, pretty much all of the time.

If you're a hipster in a major city near fiber, sure, it all works great. For the rest of us, no, daily failures are the reality.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#155

Post by LeftTurnSignal »

scurvy wrote: post #122
CydeWeys wrote: post #30

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…

> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt Cost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this ou…

This issue is almost the same issue I had when I had an outage that took down the internet connection for a local govt I worked at.

The issue was a single bad port on a switch about 3 hours away. It took two days of downtime for them to figure it out.

Now the place I work at just deals with having fibre cut 2-3 hours away and losing a days worth of work for it. At least now I don't have to rely on insiders to tell me what actually is going on. But Centurylink does offer a 4g service for backup connections....

I'm not surprised at all with this. Business as usual for Centurylink.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#156

Post by wbl »

scurvy wrote: post #136
wbl wrote: post #127

Earlier quoted context omitted.

Solution is to get some datacenters on your network as well. I'm not sure why that didn't happen before.

How would a rural WISP "get some datacenters" on its network? Sorry, I'm not grasping your previous comment.

If you have the problem that peering is getting very one-sided, the solution is to get the other side as well. So a rural ISP should try to expand into an area with cheap power and offer low-cost hosting and bandwidth to attract servers.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#157

Post by wpietri »

scurvy wrote: post #135
wpietri wrote: post #130

Earlier quoted context omitted.

We don't, and apparently they didn't either. But it was their job to do so. I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause. We can't know, of course. But I think it would be hard to argue that this is optimal, and that no…

It's hard to work on network problems when the management network is also having problems. That requires dispatching people locally to sites to see what's going on. You're blind in both eyes, not just one. Look, I'm not defending them or their choices. I'm just saying that this more more nuanced than "They suck they should have known better or spent more money." When you run a network as large as this and with as man…

I agree it's hard. Anything interesting is.

But your last paragraph specifically describes technical debt as part of the problem. If even the best org (NTT) has technical debt and CenturyLink isn't the best org, then I think it's safe to suspect that a) CenturyLink has significant technical debt, and b) they have underinvested in that technical debt compared with industry best practices.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#158

Post by icedchai »

packet_nerd wrote: post #153
icedchai wrote: post #133

Earlier quoted context omitted.

You are technically correct. However, the term “Ethernet packet” is commonly used colloquially... it does not seem worth arguing about.

Yeah, but the point is you wouldn't expect a competent network engineer to use the words "packet frames". And incompetence would seem to fit with the facts of the case, i.e. 911 was down for 2 days.

It was probably down because the only person who knew how to use tcpdump (or equivalent) was on vacation. ;)

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#159

Post by loeg »

jandrese wrote: post #152
loeg wrote: post #87

Earlier quoted context omitted.

6rd is pretty awful, generally, and I couldn't get Centurylink's to work last time I tried. I appreciate the notes and will take a look, thank you. (One big problem I've experienced in the past is that any time you configure both IPv6 and IPv4, all programs prefer IPv6. But tunneled IPv6 has much worse performance than the native IPv4, so you really want applications to prefer IPv4 in a tunneled v6 environment.)

I still use Hurricane Electric tunnels fir my v6 and ping times/throughout are so close as to be in the noise for most everything. It probably helps that I'm using a tunnel endpoint literally 10 miles from my house however.

My IPv4 ping times to Google and Facebook are 2.6 milliseconds. I suspect any HE tunnel is going to be worse than that (in a statistically significant sense). I don't know that latency would necessarily be perceptible the majority of the time, but I strongly suspect outliers would be worse and more frequent.

Also, I doubt HE's ipv6 tunnel is going to carry anything close to 1Gbps for me for free ;-).

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#160

Post by jdhzzz »

bjt wrote: post #93
JumpCrisscross wrote: post #85

Earlier quoted context omitted.

> Seems strange how bad ISP's are in the US compared to the UK UK average fixed-line download speeds in Q4 2017 were about 26 Mbps [1]. The same Q2/Q3 2018 statistic for the United States was over 96 [2]. American broadband is crappy. But in mean technological leadership, it’s ahead of the UK. (At the leading edge, I get 400/35 for $80 in Manhattan.) [1] http://www.speedtest.net/reports/united-kingdom/#fixed [2] http…

I wish they'd do the median rather than the mean. Having a small percentage of people with super high speeds will drag the mean up, but it doesn't change the experience for the typical consumer.

Bill Gates walks into a bar. The mean net worth of bar patrons is > $1,000,000,000.
Post reply on HN