Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 technician's misstep causes largest telephone outage ever reported

fiercetelecom.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#31

Post by mannykannot »

yjftsjthsd-h wrote: post #29
GrumpyNl wrote: post #26

Earlier quoted context omitted.

Its is exactly doing what you tell it to do, update without filter, it updates all.

One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.

...until one thinks it through and realizes that this would require that special value as an extra condition for every attribute in every relation mentioned in the query, which in turn would eliminate both the theoretical elegance of the relational model and the many very practical benefits that arise from it. A secondary problem is what to do if '*' is a valid value... Ad-hoc solutions are often more difficult than they first seem.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#32

Post by jlgaddis »

greenleafjacob wrote: post #8

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

Four minutes to realize SHTF, but a while longer to figure out why:

> Level 3 was aware it had a problem within four minutes, the FCC report said. The problem was difficult to diagnose, however, because no one at Level 3 was aware of the consequences of leaving that particular field empty, nor had anyone at the company previously seen the system behave the way it was behaving.

That is, they didn't know that leaving that field blank is what caused the S to HTF.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#34

Post by irishsultan »

atkbrah wrote: post #27
scotty79 wrote: post #23

I always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.

It's like one of the most common user errors with cisco cli. When you want to add an interface to a vlan you would type: switchport trunk allowed vlan add $vlan But if you by accident omit add keyword you would replace all interface vlans with $vlan switchport trunk allowed vlan $vlan

At least in that case the important keyword isn't placed at the end (i.e. you would get an error if you merely typed ` switchport trunk allowed vlan add`)

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#35

Post by aexaey »

atkbrah wrote: post #27
scotty79 wrote: post #23

I always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.

It's like one of the most common user errors with cisco cli. When you want to add an interface to a vlan you would type: switchport trunk allowed vlan add $vlan But if you by accident omit add keyword you would replace all interface vlans with $vlan switchport trunk allowed vlan $vlan

...and, if especially unlucky, that would be the trunk you're using to connect to that switch.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#36

Post by aexaey »

yjftsjthsd-h wrote: post #29
GrumpyNl wrote: post #26

Earlier quoted context omitted.

Its is exactly doing what you tell it to do, update without filter, it updates all.

One would expect to have to specify that with an asterisk or something and have blank just give a syntax error.

Percent, rather than asterisk.

  UPDATE t SET foo='bar' WHERE baz LIKE '$INPUT';
This will do no changes when $INPUT is empty, and only with $INPUT set to '%' will it update every record.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#37

Post by cesarb »

ashleyn wrote: post #25
virtuowl wrote: post #3

This sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do

My thoughts exactly: "The network management software interpreted the empty field as a 'wildcard,' meaning that the software understood the blank field as an instruction to block all calls, instead of as a null entry. This caused the switch to block calls from every number in Level 3’s non-native telephone number database." WTF kind of crappy design is that?

Probably a filtering interface with several fields, in which you fill the ones you want to filter on. If you don't fill a field, it's not used as a criteria for filtering (so empty fields are a "don't care" for that field's criteria).

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#38

Post by Already__Taken »

mdip wrote: post #12
Mtinie wrote: post #9

Earlier quoted context omitted.

This is a painful reminder for those of us who routinely design form-based interfaces to ensure ambiguous fields are explicitly understood by the user before entry or called out during verification of the submission.

An "Are you sure?" prompt goes a long way and sometimes takes a really long time to get added, even when it's historically been a problem[0]. Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had managemen…

We had some remote desktop software that asked "are you sure" about all kinds of things. If you hit logoff/restart/shutdown on a group with nothing selected it'd ask "Are you sure" yes/no and select everything in the group and perform said action. I pushed for ages to get that "default do everything" behaviour removed entirely.

If you think "are you sure" is a good solution, you may have problems much further back in the applications flow, any it may not be helping even if you do add it.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#39

Post by paulie_a »

Shivetya wrote: post #4

that is an impressive flaw. why would you ever assume a wild card across all parts of a phone number? I get a lot of spam calls where there area code + prefix matches my number and I just ignore them. I have a hard time justifying even wildcard values for the last four digits in whole, partial wild card yes to get a PBX or such

I rarely answer calls from numbers I don't recognize anymore, unless it is to simply fuck with the scammer and waste their time.

Re: Level 3 technician's misstep causes largest telephone outage ever reported

#40

Post by Mtinie »

Already__Taken wrote: post #38
mdip wrote: post #12

Earlier quoted context omitted.

An "Are you sure?" prompt goes a long way and sometimes takes a really long time to get added, even when it's historically been a problem[0]. Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had managemen…

We had some remote desktop software that asked "are you sure" about all kinds of things. If you hit logoff/restart/shutdown on a group with nothing selected it'd ask "Are you sure" yes/no and select everything in the group and perform said action. I pushed for ages to get that "default do everything" behaviour removed entirely. If you think "are you sure" is a good solution, you may have problems much further back in…

You bring up a fair point.

Excessive prompts are equally dangerous. “Click fatigue” is a real thing and you can quickly shift from a state of “let me know when something differs from the expectations” to “why are you making me agree to everything?”, which means more often than not the user just clicks blindly instead of reading the prompt for context of why this time the prompt is different.

I’m a proponent of appropriately prompting the user when their submissions, if processed, would result in ambiguous / destructive and not-easily-reversible outcomes.

Post reply on HN