Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CenturyLink 911 outage was caused by a single network card sending bad packets

twitter.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#141

Post by bogomipz »

jauer wrote: post #140
bogomipz wrote: post #139

Earlier quoted context omitted.

>"OTU also has frames." Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious. Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy. In the RFO Centurylink also mentions - "A d…

> Also do you that they would use DWDM for their management network? Yes, in the form of the Optical Supervisory Channel (OSC), which is built into DWDM gear and generally implemented as Ethernet over SONET. The OSC can also carry management traffic for other devices (aka datawire). It's Ethernet, so it has broadcasts...

Even if its OCS we're talking about the supervisory channel is truly out of band in that its generally on a proprietary wavelength isolated from your other channels carrying customer traffic.

In this sense its no different than how a copper ethernet management VLAN should not be able to take down your entire production network.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#142

Post by jauer »

bogomipz wrote: post #141
jauer wrote: post #140

Earlier quoted context omitted.

> Also do you that they would use DWDM for their management network? Yes, in the form of the Optical Supervisory Channel (OSC), which is built into DWDM gear and generally implemented as Ethernet over SONET. The OSC can also carry management traffic for other devices (aka datawire). It's Ethernet, so it has broadcasts...

Even if its OCS we're talking about the supervisory channel is truly out of band in that its generally on a proprietary wavelength isolated from your other channels carrying customer traffic. In this sense its no different than how a copper ethernet management VLAN should not be able to take down your entire production network.

"should not be able to" being the operative phrase :)

Optical control plane generally hasn't benefited from the hardening that's happened in the IP world.

Things like CoPP haven't become common practice yet.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#143

Post by loeg »

loeg wrote: post #88
gonzo wrote: post #82

Earlier quoted context omitted.

You do know that perfection with TCP over Ethernet (no 802.1q rages, no retransmits) at 1gbps is 941Mbps, right?

What is your point? All of the observed numbers I've provided are sufficiently well below that as to make the distinction between 941 and 1000 meaningless.

Come on, folks.

> Typical is 500-600 Down

500/941 vs 500/1000 is 50% vs 53% of advertised. The 3% isn't really significant. I don't think it's bad service for the price, but it certainly isn't true 1gigE.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#144

Post by gonzo »

loeg wrote: post #143
loeg wrote: post #88

Earlier quoted context omitted.

What is your point? All of the observed numbers I've provided are sufficiently well below that as to make the distinction between 941 and 1000 meaningless.

Come on, folks. > Typical is 500-600 Down 500/941 vs 500/1000 is 50% vs 53% of advertised. The 3% isn't really significant. I don't think it's bad service for the price, but it certainly isn't true 1gigE.

Here’s what you wrote

> but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#145

Post by loeg »

penagwin wrote: post #106
loeg wrote: post #48

Earlier quoted context omitted.

Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical…

IMO the plaintext part isn't a real issue. With the recent dramatic increase in https usage it won't have much impact. I'd also consider that segment "comprimised" anyway, from a security perspective anything past my modem could be MITM'ing my connecting.

Yeah, I totally agree; at the end of the day end-to-end authenticated encryption is the way to go. But not everything is end-to-end encrypted, so it's worth mentioning as a caveat. It sounds like I may be mistaken anyways and some GPON networks negotiate an AES key for downstream traffic, but I have no way to confirm that this design or implementation is secure, or that in practice CenturyLink even uses it on my local fiber.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#146

Post by bogomipz »

jauer wrote: post #142
bogomipz wrote: post #141

Earlier quoted context omitted.

Even if its OCS we're talking about the supervisory channel is truly out of band in that its generally on a proprietary wavelength isolated from your other channels carrying customer traffic. In this sense its no different than how a copper ethernet management VLAN should not be able to take down your entire production network.

"should not be able to" being the operative phrase :) Optical control plane generally hasn't benefited from the hardening that's happened in the IP world. Things like CoPP haven't become common practice yet.

>"Optical control plane generally hasn't benefited from the hardening that's happened in the IP world"

OK, but I imagine we can probably both agree that proper network design is orthogonal to the pace of development in optical transmission gear ;)

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#147

Post by loeg »

gonzo wrote: post #144
loeg wrote: post #143

Earlier quoted context omitted.

Come on, folks. > Typical is 500-600 Down 500/941 vs 500/1000 is 50% vs 53% of advertised. The 3% isn't really significant. I don't think it's bad service for the price, but it certainly isn't true 1gigE.

Here’s what you wrote > but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps

Yes, I wrote that as well. What comment are you making?

Typical observed speeds down, which is what everyone cares about for residentical internet, are 500-600 Mbps — far below 1Gbps. I mentioned the outliers for color but they're either not useful to me (I don't need 900 Mbps upload) or not representative of real service delivered (I've only observed 900 Mbps once; as I said, 500-600 is much more typical). Off-peak is typically around 700 Mbps down.

So I still don't know what comment or point you were trying to make by claiming GigE TCP can only reach 940 Mbps. It clearly isn't a counterargument to my claim that Centurylink's $80/mo, residential "gigabit" fiber offering does not deliver 1Gbps internet service (nor would I expect it to).

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#148

Post by kiallmacinnes »

topranks wrote: post #57
loeg wrote: post #49

Earlier quoted context omitted.

It is really unfortunate that the checksum is on the opposite end of the frame from the routing information on Ethernet frames.

Why? If it was at the start it’d remain useless until the whole frame was received anyway. Makes sense to have it at the end.

It also allows you to calculate the checksum as your serialising the packet data onto the send buffer, without having to get the whole packet in memory, checksum it, write the checksum, then finally write the packet data.

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#149

Post by oarsinsync »

drcross wrote: post #102
jandrese wrote: post #33

Earlier quoted context omitted.

I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes... I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on b…

>I have a very low opinion of STP. That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.

> until we discovered it's successor

Which is?

Re: CenturyLink 911 outage was caused by a single network card sending bad packets

#150

Post by insaneirish »

oarsinsync wrote: post #149
drcross wrote: post #102

Earlier quoted context omitted.

>I have a very low opinion of STP. That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.

> until we discovered it's successor Which is?

Ensuring all layer 2 domains are loop free by design and moving all redundancy to layer 3 and routing protocols.
Post reply on HN