Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 technician's misstep causes largest telephone outage ever reported

fiercetelecom.com

Post reply on HN
Print view View original

51–60 of 85 posts

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#51

Post by NateyJay »

occams_chainsaw wrote: post #50
grecy wrote: post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

> Even people that still have 2-way radios to use as phones were out of luck How would a telecom outage impact two-way radios?

Two way radios can often connect to the telephone network through a bridge at the radio repeater

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#53

Post by grecy »

NateyJay wrote: post #51
occams_chainsaw wrote: post #50

Earlier quoted context omitted.

> Even people that still have 2-way radios to use as phones were out of luck How would a telecom outage impact two-way radios?

Two way radios can often connect to the telephone network through a bridge at the radio repeater

What he said

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#54

Post by moftz »

occams_chainsaw wrote: post #50
grecy wrote: post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

> Even people that still have 2-way radios to use as phones were out of luck How would a telecom outage impact two-way radios?

There are systems used by first responders that allow for radio basestations to be connected to landlines to run to other basestations. This allows you to create more coverage area for your network. The radios run a trunked system that is more like how cellphones work than how something like CB radios work.

This is why things like ARES (Amateur Radio Emergency Service) is important for disasters where centralized systems aren't working but long range, simplex (decentralized) nets can still operate.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#55

Post by hn20180220 »

greenleafjacob wrote: post #8

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

At most organizations change management is nothing but bureaucracy that does nothing but preventing things from getting done.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#56

Post by organsnyder »

woliveirajr wrote: post #28

> The FCC report said Level 3 subsequently adopted measures to prevent a recurrence of the problem - measures in accord with best practices. What is the "Best practices" to prevent someone from leaving a blank field, since this field was interpreted as a "*" and blocked everyone ? Would a "send an e-mail telling to never leave this field blank again" enought ?

I would hope that they actually added input validation to prevent this from occurring in the future.

But it wouldn't surprise me if someone managed to consider sending a "don't do this" email with the high-priority flag set as a "best practice"...

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#57

Post by gmueckl »

archon wrote: post #30
mdip wrote: post #10

Earlier quoted context omitted.

I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…

> Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely, workstation/server. I find seeing this mentioned oddly comforting. I write my worst soft…

This is the difference between tinkering, experimentation and engineering. All of these have their proper place. There is no shame in having a shoddy piece of code as long as it is within a lab environment. Trouble usually starts when this kind of code changes hands and finds serious users.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#58

Post by blacksmith_tb »

mdip wrote: post #10
virtuowl wrote: post #3

This sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do

I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…

What keeps most of from 'shipping' our home-grade, hacked-together code to anyone else is the fear of having to support it, though. "OK, grandma, now shell into the RPi and tell me what crontab -l says..." - you would think that aside from any sort of professional ethics (or even pride) that companies in these situations wouldn't dare ship something that 'mostly worked' (unless they had superhero lawyers...)

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#59

Post by twic »

mdip wrote: post #7

I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…

> There's crossover between the rubber tires and fiber optics, apparently

Nokia also started as a rubber company:

https://en.wikipedia.org/wiki/History_of_Nokia

They got into telecommunications by buying a cable works. I don't know if that was just random empire-building, or if there are material connections between cables and rubber. I suppose you need a lot of rubber to insulate the cables.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#60

Post by abandonliberty »

avs733 wrote: post #48
grecy wrote: post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

This type of stuff is terrifying to me. When these incidents occur, people freak out and get angry and demand action...for a while. Then it gets forgotten about. As I've gotten older I've started to separate process driven organizations from progress driven organizations. Process driven ones tend to be very boring, and it can be abused, but good processes trump progress to me everyday.

>Then it gets forgotten about.

Then consumers don't want to pay for it and action ceases. You can play those odds for quite a while.

Post reply on HN