Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 technician's misstep causes largest telephone outage ever reported

fiercetelecom.com

Post reply on HN
Print view View original

21–30 of 85 posts

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#21

Post by Mtinie »

mdip wrote: post #12
Mtinie wrote: post #9

Earlier quoted context omitted.

This is a painful reminder for those of us who routinely design form-based interfaces to ensure ambiguous fields are explicitly understood by the user before entry or called out during verification of the submission.

An "Are you sure?" prompt goes a long way and sometimes takes a really long time to get added, even when it's historically been a problem[0]. Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had managemen…

I agree completely. As a designer I fear I’ve missed something like this any time a feature or function I’ve worked on is released.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#22

Post by bonesss »

greenleafjacob wrote: post #8

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

That's the conclusion of the report: no one was previously aware that an "unrelated" field in an "unrelated" activity would have production-wide consequences.

The detected an issue within 4 minutes, but it took an hour and a half to diagnose the issue, find the unwanted and unforeseen change, and revert it. That's not an absurd amount of time for a complex system with millions of users.

I would hazard to guess that this long-running system isn't even taken in as part of the companies 'change management' routines, in so far as it's job was number filtering and the operation was 'routine'. At my work we produce production change reports for production changes, but we don't fill them out every time we run a stable application...

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#24

Post by madeofpalk »

davidkuhta wrote: post #11
bigiain wrote: post #5

Earlier quoted context omitted.

Yeah, but nobody ever seems to throw the UX designers under the bus, right?

Well there was that one test missile notification system in Hawaii with stellar UX and UI...

...and it was the technician/operator that was fired.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#25

Post by ashleyn »

virtuowl wrote: post #3

This sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do

My thoughts exactly:

"The network management software interpreted the empty field as a 'wildcard,' meaning that the software understood the blank field as an instruction to block all calls, instead of as a null entry. This caused the switch to block calls from every number in Level 3’s non-native telephone number database."

WTF kind of crappy design is that?

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#26

Post by GrumpyNl »

scotty79 wrote: post #23

I always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.

Its is exactly doing what you tell it to do, update without filter, it updates all.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#27

Post by atkbrah »

scotty79 wrote: post #23

I always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.

It's like one of the most common user errors with cisco cli. When you want to add an interface to a vlan you would type:

  switchport trunk allowed vlan add $vlan
But if you by accident omit add keyword you would replace all interface vlans with $vlan

  switchport trunk allowed vlan $vlan

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#28

Post by woliveirajr »

> The FCC report said Level 3 subsequently adopted measures to prevent a recurrence of the problem - measures in accord with best practices.

What is the "Best practices" to prevent someone from leaving a blank field, since this field was interpreted as a "*" and blocked everyone ?

Would a "send an e-mail telling to never leave this field blank again" enought ?

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#29

Post by yjftsjthsd-h »

GrumpyNl wrote: post #26
scotty79 wrote: post #23

I always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.

Its is exactly doing what you tell it to do, update without filter, it updates all.

One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#30

Post by archon »

mdip wrote: post #10
virtuowl wrote: post #3

This sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do

I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…

> Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely, workstation/server.

I find seeing this mentioned oddly comforting. I write my worst software for myself. Zero validation, very little error handling, unchecked assumptions all over the code.

At least I'm not the only one out there with a barely-stable home lab setup because of shoddy programming.

Post reply on HN