Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…
Level 3 technician's misstep causes largest telephone outage ever reported
71–80 of 85 posts
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#72Earlier quoted context omitted.
I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…
To [0], would there be any value to stealing the cards not to sell them but to reverse engineer them to find vectors for attacks? "Effectively unsupported, never updated, and responsible for a massive amount of telecommunications" sounds like a great time for an attacker. Also seems like it would have value for industrial espionage.
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#73Earlier quoted context omitted.
I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…
What keeps most of from 'shipping' our home-grade, hacked-together code to anyone else is the fear of having to support it, though. "OK, grandma, now shell into the RPi and tell me what crontab -l says..." - you would think that aside from any sort of professional ethics (or even pride) that companies in these situations wouldn't dare ship something that 'mostly worked' (unless they had superhero lawyers...)
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#74I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…
Three things. 1. Thanks for the awesome TIL 2. Rumors fly about old versions of Windows, OS/2, etc still being actively used. I like to pin down and file away usage/year correlations, where possible. What sort of timeframe (roughly) was Win98 in active use here? 3. Regarding [1], I have an ancient [runs downstars to check] Compaq Prosignia 300 server here and I discovered in the (DR-DOS based) BIOS at one point that…
2. The last I had heard about that machine, specifically, was around 2011 and I'm fairly certain it was there when they eliminated the NOC in Detroit which was around 2013, if memory serves. The thing is, I would be surprised if it was actually gone.
3. Nice - I was well known was the guy who could fix anything over at Level 3 around the IT side of the house. Around 2014, I was asked to take a look at an ancient desktop PC that had been sitting in the server room at another building. It had failed weeks prior, had a modem plugged in and was discovered to have been used for a billing purpose that was apparently costing well into the 6 figures. The VP asked me if I would spare a moment to see if I could figure it out, despite it having nothing to do with my normal duties. It was dusty as heck and wouldn't boot -- nobody on the server team could get it functioning. I noticed a home-made label stuck to the top and recognized it as drive geometry. The little battery had failed, probably 15 years prior, and someone put a label on it figuring that people would understand exactly what it was. I was the only one who remembered what that was. When I made it boot, people stared at me like I was a magician. :)
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#75Earlier quoted context omitted.
Yeah, but nobody ever seems to throw the UX designers under the bus, right?
TLDR; Ill-conceived design is only part of the problem. —— You are assuming the entry form was put together with the assistance of a UX designer. In my experience, back office or configuration software rarely is seen as important enough by the Powers-That-Be to justify all the “extra research and design effort” required. Usually work like that is tossed over to whichever mid-level software engineer has an extra cycle…
I strongly recommend:
https://deardesignstudent.com/a-designers-code-of-ethics-f4a...
and:
https://medium.com/@monteiro/designs-lost-generation-ac72895...
from someone who's done _way_ more thinking about these sorts of issues than I have...
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#76Earlier quoted context omitted.
One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.
...until one thinks it through and realizes that this would require that special value as an extra condition for every attribute in every relation mentioned in the query, which in turn would eliminate both the theoretical elegance of the relational model and the many very practical benefits that arise from it. A secondary problem is what to do if '*' is a valid value... Ad-hoc solutions are often more difficult than…
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#77Well, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck.…
> Even people that still have 2-way radios to use as phones were out of luck How would a telecom outage impact two-way radios?
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#78Earlier quoted context omitted.
I can imagine it once they found out the cause. "Really? That was it? Are you fucking serious?"
sounds like most of the manufacturing issues I have been involved in... 'why did the machine break?' 'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on th…
I'm guessing I can't read the source to this particular fault tree, but I wonder where I might find others. Preferably without digging through e.g. troves of court documents and the like.
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#79Earlier quoted context omitted.
Its is exactly doing what you tell it to do, update without filter, it updates all.
One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.
Re: Level 3 technician's misstep causes largest telephone outage ever reported
#80Earlier quoted context omitted.
sounds like most of the manufacturing issues I have been involved in... 'why did the machine break?' 'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on th…
Nice. I'm guessing I can't read the source to this particular fault tree, but I wonder where I might find others. Preferably without digging through e.g. troves of court documents and the like.