Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 technician's misstep causes largest telephone outage ever reported

fiercetelecom.com

Post reply on HN
Print view View original

41–50 of 85 posts

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#41

Post by Dodgeit »

mdip wrote: post #7

I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…

Having ongoing and massive issues with a Cisco phone system where I work currently, so I was reading into it as being a Cisco system and not being surprised.

Although agree that Cisco is usually pretty solid.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#42

Post by kazen44 »

aexaey wrote: post #35
atkbrah wrote: post #27

Earlier quoted context omitted.

It's like one of the most common user errors with cisco cli. When you want to add an interface to a vlan you would type: switchport trunk allowed vlan add $vlan But if you by accident omit add keyword you would replace all interface vlans with $vlan switchport trunk allowed vlan $vlan

...and, if especially unlucky, that would be the trunk you're using to connect to that switch.

Which is why you usually have a seperate OOB network for logging into switches and routers.

If it's very mission critical, even an entire seperate cable infrastructure for OOB is not uncommon.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#43

Post by Dodgeit »

jlgaddis wrote: post #32
greenleafjacob wrote: post #8

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

Four minutes to realize SHTF, but a while longer to figure out why: > Level 3 was aware it had a problem within four minutes, the FCC report said. The problem was difficult to diagnose, however, because no one at Level 3 was aware of the consequences of leaving that particular field empty, nor had anyone at the company previously seen the system behave the way it was behaving. That is, they didn't know that leaving t…

I can imagine it once they found out the cause.

"Really? That was it? Are you fucking serious?"

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#44

Post by grecy »

Well, I dunno.

When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days.

When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.

There is no other provider for every single one of those customers, so not a single person had a single shred of connectivity.

In the larger cities they posted Police and EMT Personnel on street corners in case people needed 911.

That was a rather big outage, caused by a backup generator not starting during a power outage. We were still trying to bring systems back online a week later that had never been shutdown in 20+ years.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#45

Post by [deleted] »

kazen44 wrote: post #42
aexaey wrote: post #35

Earlier quoted context omitted.

...and, if especially unlucky, that would be the trunk you're using to connect to that switch.

Which is why you usually have a seperate OOB network for logging into switches and routers. If it's very mission critical, even an entire seperate cable infrastructure for OOB is not uncommon.

[deleted]

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#46

Post by cesarb »

exikyut wrote: post #18
mdip wrote: post #7

I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…

Three things. 1. Thanks for the awesome TIL 2. Rumors fly about old versions of Windows, OS/2, etc still being actively used. I like to pin down and file away usage/year correlations, where possible. What sort of timeframe (roughly) was Win98 in active use here? 3. Regarding [1], I have an ancient [runs downstars to check] Compaq Prosignia 300 server here and I discovered in the (DR-DOS based) BIOS at one point that…

> and I discovered in the (DR-DOS based) BIOS at one point that the serial port's electrical behavior can be customized between being edge triggered or level triggered.

Things like that make John Titor's IBM 5100 story slightly more believable. Old computers often have strange features which are omitted from their successors due to being rarely used (for instance, the original IBM PC could use current loop on its serial port connector). One could imagine someone with access to a time machine taking a short trip to the past just to grab one of these old computers as a replacement for a failed one.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#47

Post by pbhjpbhj »

atkbrah wrote: post #27
scotty79 wrote: post #23

I always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.

It's like one of the most common user errors with cisco cli. When you want to add an interface to a vlan you would type: switchport trunk allowed vlan add $vlan But if you by accident omit add keyword you would replace all interface vlans with $vlan switchport trunk allowed vlan $vlan

IIRC that's like "groupadd" on Linux distros, if you don't add a switch it replaces all that users groups.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#48

Post by avs733 »

grecy wrote: post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

This type of stuff is terrifying to me. When these incidents occur, people freak out and get angry and demand action...for a while. Then it gets forgotten about.

As I've gotten older I've started to separate process driven organizations from progress driven organizations. Process driven ones tend to be very boring, and it can be abused, but good processes trump progress to me everyday.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#49

Post by avs733 »

Dodgeit wrote: post #43
jlgaddis wrote: post #32

Earlier quoted context omitted.

Four minutes to realize SHTF, but a while longer to figure out why: > Level 3 was aware it had a problem within four minutes, the FCC report said. The problem was difficult to diagnose, however, because no one at Level 3 was aware of the consequences of leaving that particular field empty, nor had anyone at the company previously seen the system behave the way it was behaving. That is, they didn't know that leaving t…

I can imagine it once they found out the cause. "Really? That was it? Are you fucking serious?"

sounds like most of the manufacturing issues I have been involved in...

'why did the machine break?'

'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on this machine.'

'well why aren't they interchangeable on this machine'

'Y reacts with the glue used to assemble the machine, which was a change in the newer versions because of EPA regulations, so doing this weekly maintenance task for 8 years was finally enough to degrade the glue'

'so why was chemical Y stored near here?'

'because X has to be kept so many feet from Y. Last year in the efficiency audit we found techs had to walk to far on average to get X so we moved the X cabinet, which resulted in us moving Y to here.'

'has Y been used on any of the other machines when it shouldn't have been?'

'we don't know and we aren't sure how to check'

Fault trees usually make for really interesting reading.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#50

Post by occams_chainsaw »

grecy wrote: post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

> Even people that still have 2-way radios to use as phones were out of luck

How would a telecom outage impact two-way radios?

Post reply on HN