Live data from Hacker News

Level 3 technician's misstep causes largest telephone outage ever reported

fiercetelecom.com

71–80 of 85 posts

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#71
post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

[deleted]

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#72
post #10

Earlier quoted context omitted.

I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…

To [0], would there be any value to stealing the cards not to sell them but to reverse engineer them to find vectors for attacks? "Effectively unsupported, never updated, and responsible for a massive amount of telecommunications" sounds like a great time for an attacker. Also seems like it would have value for industrial espionage.

I can't say that I'm completely sure, but I'm going to guess the answer is "Probably Not". The big telecoms tend to be very conservative about where they spend their money -- understandable with a product this expensive despite the grief one tends to have in working with the product. The market is also moving toward more generic equipment and less specialized hardware for a lot of things and I'd imagine these cards will fall into that category at some point, as well.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#73
post #10

Earlier quoted context omitted.

I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…

What keeps most of from 'shipping' our home-grade, hacked-together code to anyone else is the fear of having to support it, though. "OK, grandma, now shell into the RPi and tell me what crontab -l says..." - you would think that aside from any sort of professional ethics (or even pride) that companies in these situations wouldn't dare ship something that 'mostly worked' (unless they had superhero lawyers...)

LOL - I'm reading this as I'm checking my home nagios server to see why I just got an e-mail about the Raspberry Pi at my parent's cabin up north.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#74
post #18
post #7

I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…

Three things. 1. Thanks for the awesome TIL 2. Rumors fly about old versions of Windows, OS/2, etc still being actively used. I like to pin down and file away usage/year correlations, where possible. What sort of timeframe (roughly) was Win98 in active use here? 3. Regarding [1], I have an ancient [runs downstars to check] Compaq Prosignia 300 server here and I discovered in the (DR-DOS based) BIOS at one point that…

1. You're kindly welcome.

2. The last I had heard about that machine, specifically, was around 2011 and I'm fairly certain it was there when they eliminated the NOC in Detroit which was around 2013, if memory serves. The thing is, I would be surprised if it was actually gone.

3. Nice - I was well known was the guy who could fix anything over at Level 3 around the IT side of the house. Around 2014, I was asked to take a look at an ancient desktop PC that had been sitting in the server room at another building. It had failed weeks prior, had a modem plugged in and was discovered to have been used for a billing purpose that was apparently costing well into the 6 figures. The VP asked me if I would spare a moment to see if I could figure it out, despite it having nothing to do with my normal duties. It was dusty as heck and wouldn't boot -- nobody on the server team could get it functioning. I noticed a home-made label stuck to the top and recognized it as drive geometry. The little battery had failed, probably 15 years prior, and someone put a label on it figuring that people would understand exactly what it was. I was the only one who remembered what that was. When I made it boot, people stared at me like I was a magician. :)

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#75
post #19
post #5

Earlier quoted context omitted.

Yeah, but nobody ever seems to throw the UX designers under the bus, right?

TLDR; Ill-conceived design is only part of the problem. —— You are assuming the entry form was put together with the assistance of a UX designer. In my experience, back office or configuration software rarely is seen as important enough by the Powers-That-Be to justify all the “extra research and design effort” required. Usually work like that is tossed over to whichever mid-level software engineer has an extra cycle…

100% agree with everything you say here.

I strongly recommend:

https://deardesignstudent.com/a-designers-code-of-ethics-f4a...

and:

https://medium.com/@monteiro/designs-lost-generation-ac72895...

from someone who's done _way_ more thinking about these sorts of issues than I have...

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#76

Earlier quoted context omitted.

One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.

...until one thinks it through and realizes that this would require that special value as an extra condition for every attribute in every relation mentioned in the query, which in turn would eliminate both the theoretical elegance of the relational model and the many very practical benefits that arise from it. A secondary problem is what to do if '*' is a valid value... Ad-hoc solutions are often more difficult than…

This is telephony. '*' is indeed a valid element of a telephone number field.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#77
post #44

Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…

> Even people that still have 2-way radios to use as phones were out of luck How would a telecom outage impact two-way radios?

I have an ancient book on Private Mobile Radio. It has a section on using leased lines in it

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#78
post #49
post #43

Earlier quoted context omitted.

I can imagine it once they found out the cause. "Really? That was it? Are you fucking serious?"

sounds like most of the manufacturing issues I have been involved in... 'why did the machine break?' 'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on th…

Nice.

I'm guessing I can't read the source to this particular fault tree, but I wonder where I might find others. Preferably without digging through e.g. troves of court documents and the like.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#79

Earlier quoted context omitted.

Its is exactly doing what you tell it to do, update without filter, it updates all.

One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.

Ask yourself how many search forms you have filled in where you are allowed to leave the fields that you are not searching on blank. That is a fairly common design.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#80
post #78
post #49

Earlier quoted context omitted.

sounds like most of the manufacturing issues I have been involved in... 'why did the machine break?' 'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on th…

Nice. I'm guessing I can't read the source to this particular fault tree, but I wonder where I might find others. Preferably without digging through e.g. troves of court documents and the like.

https://www.ntsb.gov/investigations/AccidentReports/Pages/Ac...
Post reply on HN