Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

61–70 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#61

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source…

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server:

    lookupByCity(...) {
        ....
        try {
            conn = connectionPool.getConnection();
            stmt = conn.createStatement();
            ...
         } finally {
             if (stmt != null) {
                 stmt.close();
             }
             if (conn != null) {
                 conn.close();
             }
         }
    }
close() can throw, and in the circumstance of the outage it did for stmt, leading to the connection not getting closed and eventually the pool being exhausted with every thread blocked waiting for a connection. It's an interesting chain of failures, arguably the presence of such a chain is the real root cause, rather than the unhandled sql exception.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#62
post #7

Earlier quoted context omitted.

This is probably the same guy from fuckinglevel3.com. We used level3 for years and sent each other that link almost daily. I guess he had to update after CL bought them :] Don't expect a technical report from CL/L3. We had 60+ mpls/vpls circuits from them and all our reports were very high level. Source: am neteng

I always heard L3 was quite good. Who is if they’re not?

There is no such thing as a good national carrier right now, L3 just happened to be the least shit of the group to deal with. Your best bet is to try to find a decent local/regional carrier as your primary circuit provider and then use L3/The Devil/ATT (in that order) as your secondary. Yes your regional/local is still going to hand off to those guys but at least they'll be doing all of them and you get real support for local issues.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#63
post #59
post #40

Earlier quoted context omitted.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

I don't know if it's widespread but at least here in WA you can pay Comcast an extra $50/month and they'll remove the data cap.

Comcast Business class accounts don't have the cap as well. It's slightly more expensive and comes with slower speed tiers but the support has been better. It's not required to be a business to go through that channel.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#64
post #40
post #39

Earlier quoted context omitted.

At least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

Seems strange how bad ISP's are in the US compared to the UK.

I can get 400/35 Fiber to the Home with no datacap (no throttling either) for £54 a month (~$70).

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#65
post #48

Earlier quoted context omitted.

centurylink fiber is pretty great as well

Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical…

You sure it’s plaintext? GPON networks should be using AES128 to each ONT.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#66

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink,…

> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running:

Especially now with cordcutting becoming the norm and traditional cable subscription number dropping, large providers are scrambling to find ways to extract profit.

With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access.

Because of this we're going to start seeing a lot more data caps, neglecting basic network infrastructure, and other tactics to extract as much profit as possible from customers.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#67

Packets or frames? The report mentions both (including "packet frames", which, ugh)

These terms are interchangeable. Although typically one or other is used when speaking about a specific technology. Also see datagram, cell and probably others I’m forgetting right now.

But packets and frames as implemented are different things. One encapsulates the other.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#68
post #44
post #40

Earlier quoted context omitted.

Your cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.

This is why amazes me. At my home we blow through on average 25 GB a day in gaming, streaming video, other downloads etc. limits like these are terrible. 80 a month? Wtf?!

25 GB a day? Are you streaming 4K?

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#69

Earlier quoted context omitted.

Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.

centurylink fiber is pretty great as well

Doesn't exist across 99% of their last mile footprint. Just one or two cities.

I would guess the vast majority of their revenue comes from extorting old people and farmers with shitty overpriced ADSL that squirrels eat into and take down annually.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#70

The descriptions so far about this problem are either at 30000 foot vagueness or they're in technical shorthand that just assumes the audience is 100% pro network engineers. Don't "bad packets" get dropped at the first switch? Isn't that one of the main benefits of packet based switching? Was this even an ethernet packet or something else like an optical transport protocol (eg OTN)?

They generally only get dropped if they have an invalid checksum. Since checksums are hardware accelerated, the invalid packet probably had a valid checksum applied to it.

In addition, a valid checksum is no guarantee that the packet is valid. They are very weak hashes, not cryptographic. If a NIC goes haywire and sends "random" data at wire speed, there will be bad packets with valid checksums.
Post reply on HN