CenturyLink 911 outage was caused by a single network card sending bad packets • Hacker News Board

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#71

Post by Trisell » Sun, Dec 30, 2018, 10:17 PM UTC

Trisell wrote: post #39

jandrese wrote: post #31

Earlier quoted context omitted.

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

At least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.

30gb to 1.5tb. And the last one I had it I was paying over $100 a month for 150gb.

https://support.cableone.net/hc/en-us/articles/115009159707-...

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#72

Post by jonathankoren » Sun, Dec 30, 2018, 10:25 PM UTC

noir_lord wrote: post #64

sethdj wrote: post #40

Earlier quoted context omitted.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

Seems strange how bad ISP's are in the US compared to the UK. I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

What's really strange is how the crappy broadband is continuously justified in the US.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#73

Post by toufiqbarhamov » Sun, Dec 30, 2018, 10:35 PM UTC

jonathankoren wrote: post #72

noir_lord wrote: post #64

Earlier quoted context omitted.

Seems strange how bad ISP's are in the US compared to the UK. I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

What's really strange is how the crappy broadband is continuously justified in the US.

Replace “broadband” with everything from education and healthcare, through infrastructure and a dozen other words, and you’re still right. I don’t understand it either.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#74

Post by cpeterso » Sun, Dec 30, 2018, 10:41 PM UTC

Jach wrote: post #61

carlsborg wrote: post #8

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); }…

"Release It" is an awesome book. I contributed a couple FindBug Java checkers to warn about some of the problems the book describes.

The case you mention above might have been prevented by using checked Java exceptions. Our programming languages and tools could be doing a lot more to catch these problems at compile time or make them impossible by language or API design.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#75

Post by X-Istence » Sun, Dec 30, 2018, 10:50 PM UTC

sethdj wrote: post #40

Trisell wrote: post #39

Earlier quoted context omitted.

At least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

I live alone, and generally I do about ~400GB of data transfer on Comcast a month. I work from home, and spend a lot of time downloading Docker containers and other such things to do my work.

Last month apparently I hit a new high for me: 890GB. Lots of devices needing updates/games downloaded...

I am really surprised that with a 7 person household you don't go through your data even faster.

I even run a local caching server for all my Apple devices, which over the last 30 days has saved me about ~50GB from having to be served from origin, with about ~100GB being served to clients.

Although I tend to also be a heavy user of Apple's iTunes and particularly their movies service, so I have a feeling a lot of that cached data is me putting movies on in the background and them being streamed from the cache rather than from Apple directly.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#76

Post by X-Istence » Sun, Dec 30, 2018, 10:53 PM UTC

kijiki wrote: post #38

extrapickles wrote: post #27

Earlier quoted context omitted.

They generally only get dropped if they have an invalid checksum. Since checksums are hardware accelerated, the invalid packet probably had a valid checksum applied to it.

If cut-through is enabled in the switch, it won't even drop on bad checksum, since by the time the switch can tell the checksum is wrong, it has already forwarded the entire packet.

It'll just log it... and then you get to play "let's find where it's coming from" as each switch just forwards that bad boy on as fast as possible.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#77

Post by packet_nerd » Sun, Dec 30, 2018, 11:05 PM UTC

tyingq wrote: post #15

carlsborg wrote: post #8

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

One example might be "ip helper" setups in a Cisco router. People use it to forward DHCP requests over a WAN. One broken SIP device, no flood control, and...bam, your WAN is hosed.

Ip helper usually only shows up in the context of DHCP so its easy to forget it forwards a lot more than just DHCP broadcasts.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#78

Post by bch » Sun, Dec 30, 2018, 11:07 PM UTC

hinkley wrote: post #41

bch wrote: post #21

Earlier quoted context omitted.

Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...

The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s. There are lots of proximate reasons that a failure can avoid all of your sanity checks, but the simplest is when your organization is a little insane. I try to steer the conversation toward avoiding surprises, other times toward ergonomics (don’t rely so much on humans in the moment to do t…

> The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s.

Agreed. I think it’s a case for running a Joy Driven Development operation first, and looking to organizational responsibility first (versus looking for somewhere to cast blame), but practically, that probably too utopian to bear out in practice. No reason not to keep it in mind, though.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#79

Post by ransford » Sun, Dec 30, 2018, 11:09 PM UTC

loeg wrote: post #48

waynesonfire wrote: post #36

Earlier quoted context omitted.

centurylink fiber is pretty great as well

Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical…

I've been getting perfectly adequate IPv6 via their 6rd gateway (on fiber, in Seattle). I wrote some nerd notes on using one's own wifi router and setting up IPv6: http://b.tra.in/2015/07/notes-on-centurylink-fiber.html

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#80

Post by cyberpunk » Sun, Dec 30, 2018, 11:18 PM UTC

linker3000 wrote: post #14

dec0dedab0de wrote: post #9

Earlier quoted context omitted.

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

I used to look after an office location that had three racks with older generation HP Procurve switches. About once every 4-6 months, the whole lot would go haywire and the only way to settle down the network was to switch them all off, wait, and then power up again. We never did track down the root cause and could never make it happen on demand,

Oh god, I had to deal with some procurve switches in a past life. They were a nightmare to manage and unreliable.

We actually ended up replacing them on 2 racks with 24port unmanaged gbit 3com ones we bought from some high street electronics shop one night during an outage... Not a fun memory..

(Of course, those were later replaced with some proper equipment)