If you changed every bit individually it wouldn't have broken... so really, 1 bit broke the network.
How four packets broke CenturyLink's network
11–20 of 28 posts
Re: How four packets broke CenturyLink's network
#12> As to what can be done to prevent similar failures, the FCC is recommending CenturyLink and other backbone providers take some basic steps, such as disabling unused features on network equipment, installing and maintaining alarms that warn admins when memory or processor use is reaching its peak, and having backup procedures in the event networking gear becomes unreachable. Disabling unused services? Alarms when ne…
It's kind of funny. These are best practices for running basic run of the mill web services, even something like a forum or personal homepage. Admittedly there's an obvious, massive difference in complexity, but you would expect the gold standard best practices to come from something mission critical like core Internet services and flow down to less critical services, not the other way around.
Well it is easy to find time to add gold-plating to a small basically useless service, but those guys are probably swamped or try to cut cost by being agile or something.
Re: How four packets broke CenturyLink's network
#13What protocol is that? Optional TTL sounds like the really fatal part.
Assuming that by > 3. no expiration time, meaning that the packet would not be dropped for being created too long ago; and they mean the TTL was set to zero. From RFC 1812: > A router MUST NOT originate or forward a datagram with a Time-to-Live (TTL) value of zero. So a packet with a TTL=0 should never be on the wire (Example a router receives a packet with TTL=1, if it's not destined for that specific router, then i…
Reading Infinera's network brochure, https://www.infinera.com/wp-content/uploads/Infinera-DTN-X-F..., they are talking about terabit speeds over fiber. I doubt they are using the Internet Protocol or anything close. I mean, they could be (https://en.wikipedia.org/wiki/IPoDWDM), but they have a bunch of different communication protocols going over it. I saw MPLS (https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching) on Twitter and that has a TTL too, but unfortunately the FCC report doesn't go into detail. It's only slightly more informative than the outage report from last year: https://twitter.com/briankrebs/status/1079135599309791235/ph...
Re: How four packets broke CenturyLink's network
#14Earlier quoted context omitted.
It's kind of funny. These are best practices for running basic run of the mill web services, even something like a forum or personal homepage. Admittedly there's an obvious, massive difference in complexity, but you would expect the gold standard best practices to come from something mission critical like core Internet services and flow down to less critical services, not the other way around.
Well it is easy to find time to add gold-plating to a small basically useless service, but those guys are probably swamped or try to cut cost by being agile or something.
Cutting costs by fighting fires all the time^W^W^W^W^Wremoving smoke detectors. The classic strategy.
Re: How four packets broke CenturyLink's network
#15Network engineer here: clink bridged all of the management controllers on their infinera dwdm shelves together into one multi state sized L2 broadcast domain. Best guess is because it made them easier to SNMP poll and to run other management tools to admin them.
Within the circle of people who really know what went on, we've been laughing at them for months.
Re: How four packets broke CenturyLink's network
#16Network engineer here: clink bridged all of the management controllers on their infinera dwdm shelves together into one multi state sized L2 broadcast domain. Best guess is because it made them easier to SNMP poll and to run other management tools to admin them. Within the circle of people who really know what went on, we've been laughing at them for months.
The oddest part was that FFC recommendations didn’t include “limit the size of your broadcast domain”.
Large flat L2 are a classic time bomb, with their builders proudly exclaiming “look ma, no hands!” until they painfully get reminded of their mistake :)
Re: How four packets broke CenturyLink's network
#17Network engineer here: clink bridged all of the management controllers on their infinera dwdm shelves together into one multi state sized L2 broadcast domain. Best guess is because it made them easier to SNMP poll and to run other management tools to admin them. Within the circle of people who really know what went on, we've been laughing at them for months.
I know some of these words.
Re: How four packets broke CenturyLink's network
#18Is this a broadcast storm?
Re: How four packets broke CenturyLink's network
#19Network engineer here: clink bridged all of the management controllers on their infinera dwdm shelves together into one multi state sized L2 broadcast domain. Best guess is because it made them easier to SNMP poll and to run other management tools to admin them. Within the circle of people who really know what went on, we've been laughing at them for months.
Is there a source or is that a guess at what happened? That sounds immensely incompetent to the point that I find it hard to believe
Re: How four packets broke CenturyLink's network
#20What protocol is that? Optional TTL sounds like the really fatal part.
Assuming that by > 3. no expiration time, meaning that the packet would not be dropped for being created too long ago; and they mean the TTL was set to zero. From RFC 1812: > A router MUST NOT originate or forward a datagram with a Time-to-Live (TTL) value of zero. So a packet with a TTL=0 should never be on the wire (Example a router receives a packet with TTL=1, if it's not destined for that specific router, then i…
This could be a problem
Usually the lowest TTL on the wire is '1' - the next router then subtracts 1, the value is zero, and the packet is dropped on the same router (and icmp sent back).
If someone didn't put an aditiional if() to check, this could cause many problems, especially with broadcasts. And why would they check, if no device sends out packets like this normally (without someone else not doing an if() check, or if someone sent those packets on purpose).