Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 technician's misstep causes largest telephone outage ever reported

fiercetelecom.com

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#61

Post by nkrisc »

Already__Taken wrote: post #38
mdip wrote: post #12

Earlier quoted context omitted.

An "Are you sure?" prompt goes a long way and sometimes takes a really long time to get added, even when it's historically been a problem[0]. Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had managemen…

We had some remote desktop software that asked "are you sure" about all kinds of things. If you hit logoff/restart/shutdown on a group with nothing selected it'd ask "Are you sure" yes/no and select everything in the group and perform said action. I pushed for ages to get that "default do everything" behaviour removed entirely. If you think "are you sure" is a good solution, you may have problems much further back in…

Requiring confirmation for an action should be something a user very rarely, if ever, sees. That ensures when it does occur it 1) is serious and 2) surprises the user, gaining their full attention.

Another approach here would have been implementing sane defaults. Blank field as wildcard is not a sane default. Default to that which has the least impact.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#62

Post by nkrisc »

paulie_a wrote: post #39
Shivetya wrote: post #4

that is an impressive flaw. why would you ever assume a wild card across all parts of a phone number? I get a lot of spam calls where there area code + prefix matches my number and I just ignore them. I have a hard time justifying even wildcard values for the last four digits in whole, partial wild card yes to get a PBX or such

I rarely answer calls from numbers I don't recognize anymore, unless it is to simply fuck with the scammer and waste their time.

I never even answer to mess with them because then they know it's a good number.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#63

Post by sleepybrett »

mdip wrote: post #12
Mtinie wrote: post #9

Earlier quoted context omitted.

This is a painful reminder for those of us who routinely design form-based interfaces to ensure ambiguous fields are explicitly understood by the user before entry or called out during verification of the submission.

An "Are you sure?" prompt goes a long way and sometimes takes a really long time to get added, even when it's historically been a problem[0]. Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had managemen…

While "Are you sure?" is often a great prompt, there are two things that can be done to improve it.

1) Just putting AYS? after every prompt is bad, people stop taking the time to think and just mash Y.

2) Discussing the consequences in the AYS? is better.

For instance if you @channel in slack you get an AYS that lists the number of people you are going to piss off with the notification. Great UX... and yet people at my office still seem to think things like girl scout cookies are worth @channel'ing #general with a few thousand people in it is a good idea.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#64

Post by kjs3 »

mdip wrote: post #7

I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…

I'm betting Wellfleet/Bay Networks/Nortel "Site Manager" infrastructure management tool. Awful, awful piece of work, with exactly bugs like "well of course if you leave this field blank you want a *". You still find old Nortel gear hanging out all over the place in telco networks.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#65

Post by greenleafjacob »

gonzo41 wrote: post #15
greenleafjacob wrote: post #8

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

1 minutes to fix. 30 minutes rationalizing that the 1 minute fix your thinking of is going to work. 45 minutes convincing others. 14 minutes of terrible self doubt and worry. And we're back up. Ops life!

Mitigate first. If you are confident in your change management system then rolling back is always safe.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#66

Post by greenleafjacob »

jlgaddis wrote: post #32
greenleafjacob wrote: post #8

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

Four minutes to realize SHTF, but a while longer to figure out why: > Level 3 was aware it had a problem within four minutes, the FCC report said. The problem was difficult to diagnose, however, because no one at Level 3 was aware of the consequences of leaving that particular field empty, nor had anyone at the company previously seen the system behave the way it was behaving. That is, they didn't know that leaving t…

Figuring out why can be avoided if you have proper change management. If you are root causing every issue then that will drastically increase TTM. Instead mitigate first and just revert the change.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#67

Post by avs733 »

abandonliberty wrote: post #60
avs733 wrote: post #48

Earlier quoted context omitted.

This type of stuff is terrifying to me. When these incidents occur, people freak out and get angry and demand action...for a while. Then it gets forgotten about. As I've gotten older I've started to separate process driven organizations from progress driven organizations. Process driven ones tend to be very boring, and it can be abused, but good processes trump progress to me everyday.

>Then it gets forgotten about. Then consumers don't want to pay for it and action ceases. You can play those odds for quite a while.

unfortunately.

I forgot where I read it, but there was a story/quote/interview about how politicians can never run on maintenance because maintenance isn't sexy, they need to run on big projects.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#68

Post by greglindahl »

ashleyn wrote: post #25
virtuowl wrote: post #3

This sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do

My thoughts exactly: "The network management software interpreted the empty field as a 'wildcard,' meaning that the software understood the blank field as an instruction to block all calls, instead of as a null entry. This caused the switch to block calls from every number in Level 3’s non-native telephone number database." WTF kind of crappy design is that?

This is the basic design pattern: "Google mistakes entire web for malware"

https://www.theregister.co.uk/2009/01/31/google_malware_snaf...

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#69

Post by scotty79 »

mannykannot wrote: post #31
yjftsjthsd-h wrote: post #29

Earlier quoted context omitted.

One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.

...until one thinks it through and realizes that this would require that special value as an extra condition for every attribute in every relation mentioned in the query, which in turn would eliminate both the theoretical elegance of the relational model and the many very practical benefits that arise from it. A secondary problem is what to do if '*' is a valid value... Ad-hoc solutions are often more difficult than…

Not really. You could have a keyword ALL and UPDATE should require either WHERE or ALL.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#70

Post by cosmie »

nkrisc wrote: post #62
paulie_a wrote: post #39

Earlier quoted context omitted.

I rarely answer calls from numbers I don't recognize anymore, unless it is to simply fuck with the scammer and waste their time.

I never even answer to mess with them because then they know it's a good number.

The sheer fact that it routed to a line that rang is enough for them to know it's a good number. Their auto-dialer could be configured such that answering it increased the cadence of calls or something such as that. But the absence of the call being routed to a not-in-service/no-longer-available response is enough of a signal for them to know it's a good number.

Source: I did data management for a company that performed a high volume of outbound business dials (not consumer lines). At one point we evaluated productizing our non-valid numbers list, so that businesses could do things like flag when their main contact at an account was no longer at the company, triggering an automatic alert to follow up with the remaining contacts at the company and re-establish a relationship. CNAM lookup services like Twilio Lookup[1] don't do so well at this use case, since companies tend to reserve a full block of phone numbers (always showing as active when doing a CNAM lookup), but when an employee leaves their line will temporarily be de-activated internally until it's re-assigned to a new employee.

[1] https://www.twilio.com/lookup

Post reply on HN