Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

101–110 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#101
post #61

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…

the presence of such a chain is the real root cause, rather than the unhandled sql exception.

This is really interesting and something which bugs me about root cause analysis and it's a neat coincidence that this has been quoted relative to an aviation incident.

In aviation, incidents and accidents are investigated with the understanding that there is never a single cause of an accident. It's known as the swiss cheese model. All the holes in the swiss cheese have to line up for something to go wrong. Even in a seemingly simple "pilot error" accident, there are years of initial and recurrent training factors, ergonomic and human factors and so on which all lead to the event. It's exceedingly rare for a single "root cause" to be the whole story.

Medicine is starting to adopt techniques learned from aviation like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#102

Earlier quoted context omitted.

My reading of the tea leaves is that the invalid packets where for some low level control protocol. I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad pack…

I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes... I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on b…

>I have a very low opinion of STP.

That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#103
post #53

I once took down an entire network with a Dev node running a misconfigured DHCP server. Second time with a snmp v2 ddos. If your network isn't properly configured these things can happen easily.

> If your network isn't properly configured these things can happen easily.

Absolutely true. However, if you are an ISP, then not correctly configuring your network is... unimpressive.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#104
post #29

Makes you wonder how secure their backhaul really is? If the whole thing is a single flat logical network (one that could allow bad packets to propagate as we witnessed) that would suggest it is also quite vulnerable to malicious actions. It is all well and good applying a filter, but that seems like a bandaid fix. Why is equipment even able to talk that has no reason to do so? Seems like they've put convenience over…

Much networking equipment is not designed to handle malicious or bizarre traffic. TCP/IP is amazingly brittle, and often fails on me in surprising ways that the standards say should never ever happen.

I don't think that's fair to say. Billions of people unlock their phone or log in to their computers every morning and everything works, pretty much all of the time.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#105
Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure. A single interface shutdown command would have been all that was needed if they had remote access.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#106
post #48

Earlier quoted context omitted.

centurylink fiber is pretty great as well

Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical…

IMO the plaintext part isn't a real issue. With the recent dramatic increase in https usage it won't have much impact.

I'd also consider that segment "comprimised" anyway, from a security perspective anything past my modem could be MITM'ing my connecting.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#107
post #96

Earlier quoted context omitted.

I live alone, and generally I do about ~400GB of data transfer on Comcast a month. I work from home, and spend a lot of time downloading Docker containers and other such things to do my work. Last month apparently I hit a new high for me: 890GB. Lots of devices needing updates/games downloaded... I am really surprised that with a 7 person household you don't go through your data even faster. I even run a local cachin…

Any option to host your docker stuff on a VPS or other system that isn't using your home internet connection?

I think he/she uses Docker as a development tool on his machine, with volumes and everything.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#108
post #43

Earlier quoted context omitted.

To be fair, this does seem like an extremely rare root cause, but yes the impact was awful, so it'll be worth noting how CenturyLink responds and what they'll do to try to prevent similar issues in future.

At scale, rare errors will occur on a semi-regular basis. Multi-day outages tend to be indicative of a dysfunctional internal organization.

Totally fair - I don't mean to excuse the results. That's an interesting insight that perhaps it's possible to identify organizations who have scaled (or whose products have scaled) too quickly based on whether they can cope with outages.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#109
post #61

Earlier quoted context omitted.

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…

I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!

Yes; although in GC’ed languages like Go and JS it’s still very easy to leak OS level resources like file handles because you still need to remember to close() them. (Although go’s defer blocks are a fantastic assist here).

This is one area Rust really excels - the same mechanism for making sure memory gets cleaned up also automatically closes network sockets and file descriptors when they go out of scope. Even in the case of errors it’s impossible to forget to clean up. That entire finally block is unnecessary in rust.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#110

Earlier quoted context omitted.

I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!

Yes; although in GC’ed languages like Go and JS it’s still very easy to leak OS level resources like file handles because you still need to remember to close() them. (Although go’s defer blocks are a fantastic assist here). This is one area Rust really excels - the same mechanism for making sure memory gets cleaned up also automatically closes network sockets and file descriptors when they go out of scope. Even in th…

You're talking about the Drop trait?

I'm learning Rust coming from Go. It looks cool, but it also concerns me how most data structures in the stdlib use unsafe blocks to defeat the borrow checker. This is not the point of Rust, I would have thought?!

Post reply on HN