Level 3 technician's misstep causes largest telephone outage ever reported

Level 3 technician's misstep causes largest telephone outage ever reported
fiercetelecom.com

The article and at least one other commenter mentioned this being a UI problem, and all I can say is "bingo". They didn't identify the vendor, but the article called out Cisco. I am a little skeptical of that, personally[0]. I lean toward it being something else, mainly because of the statement that followed "no one at Level 3 was aware of the consequences of leaving that field empty". We had a lot of very knowledgeable Cisco folks there. There are a few folks that I knew personally that were probably among the top folks on administering that equipment outside of Cisco. In addition, if a problem like this arose, they're accessible and helpful.

It was almost certainly one of the many, many, outdated software applications that make up the vast array of management interfaces into the equipment. I worked inside one of the phone switches (technically, one of the test switches). About a year prior to my leaving, the room next to my desk was filled floor to ceiling with cards that had been there on the day I started. I was told these cards ran along the lines of $30,000 a piece. If you wanted to upgrade the software, you had to replace the card. So we had machines in our operations center that were on isolated networks running Windows 98 in order to run the executable required to configure the switches[1]. We had devices made by Pirelli[3] that had similarly awful software. I knew of about 5 different devices with 5 different problem management tools but there were several more. And there were the handful of devices that were well known as ones "you don't even breathe on" (and I've never heard those words said about the Cisco products ... not that people spoke terribly highly of them, but never quite that negatively).

Telecom went through a really bad time in the early 2000s. At Global Crossing, every 6 months about 10% of the staff was laid off ... this happened for 10 years. Hardware didn't get upgraded, and therefore software didn't get upgraded. Sofware support contracts and maintenance were allowed to lapse. The quality of the software -- since its audience was a small handful of companies, many of which in the same financial state as Global Crossing -- was awful. After that many layoffs, the one or two guys who knew the myriad of corner cases involved in operating some of the management interfaces were on to other jobs at other companies, or retired. That this sort of thing didn't happen somewhat regularly is still surprising to me to this day.

At the time that I left, Level 3 was doing better from a financial standpoint and money was being invested in modernising a lot of these problems, but it's a huge network with many switch sites[4] in any major city that Level 3 terminated wires. Each of those sites had an array of equipment, some of it common, some of it from companies that went under shortly after the dot-com bust.

[0] It could easily have been a Cisco product, but there are so many -- far worse -- products out there that nobody outside of telecom has ever dealt with. I lean more toward that.

[1] Before someone says "Why didn't you virtualize them?". That was done for some machines with both running inside the isolated network, but it became more of a hassle than it was worth since these machines had to occasionally connect to the devices via serial port. The very buggy software included a very temperamental driver that only worked with a few models of older PCs and IBM (not Lenovo) Thinkpad laptop serial ports ... and only then if one of the wires was cut on the cable.

[2] Earlier than the style recently featured in a post about enthusiasts for the brand upgrading the motherboards.

[3] https://www.pirelli.com/global/en-ww/homepage ... Yes, that Pirelli. There's crossover between the rubber tires and fiber optics, apparently ... that was news to me. And they had some of the worst software I've ever encountered.

[4] We had, I think, 4 in Detroit. Some are not in terribly convenient locations, either. One in the area required passing an expensive OSHA certification, wearing steel toed boots and a hard hat if you wished to access it due to its proximity to a rail yard. I'd been there once and couldn't imagine the need for the restrictions.