11–20 of 85 posts
Oh wow. > The technician left empty a field that would normally contain a target telephone number. The network management software interpreted the empty field as a 'wildcard,' ... Exactly the same type of technical error happened nine years ago at Google! > We maintain a list of [malicious] sites through both manual and automated methods. We periodically update that list and released one such update to the site this…
This is a painful reminder for those of us who routinely design form-based interfaces to ensure ambiguous fields are explicitly understood by the user before entry or called out during verification of the submission.
Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had management applications for some switch software that we had to run on Windows 98 using ThinkPad laptops[1].
[0] https://en.wikipedia.org/wiki/Rm_(Unix)#Protection_of_the_fi...
[1] I was a Frontier, then Global Crossing and ultimately Level 3 employee for around 17 years. The story about the ThinkPad Laptops is detailed in another comment in this post.
Oh wow. > The technician left empty a field that would normally contain a target telephone number. The network management software interpreted the empty field as a 'wildcard,' ... Exactly the same type of technical error happened nine years ago at Google! > We maintain a list of [malicious] sites through both manual and automated methods. We periodically update that list and released one such update to the site this…
This is a painful reminder for those of us who routinely design form-based interfaces to ensure ambiguous fields are explicitly understood by the user before entry or called out during verification of the submission.
This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?
This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?
Perhaps that would not have been possible with this style of equipment, I don’t know. But at least yes, change management procedures should have been in place and obeyed.
However, I hope they don’t fire the guy. I once read of an employee of some company who created a $600k mistake. They asked the CEO if he would fire the employee. “I just paid $600k to educate one of my employees. Do you think I’m going to give him away to my competitors?”
That has stuck with me. Unless there is clearly malicious intent I want to give second chances.
that is an impressive flaw. why would you ever assume a wild card across all parts of a phone number? I get a lot of spam calls where there area code + prefix matches my number and I just ignore them. I have a hard time justifying even wildcard values for the last four digits in whole, partial wild card yes to get a PBX or such
It sounds crazy, but consider a similar circumstance of popping something illegal into an Apache or nginx configuration file. The service fails to start and anything hosted behind it is down. I'm not saying it's acceptable, just likely[0]. The difference here is that this software has an audience of very few people, is poorly developed to begin with, and usually outputs error messages similar to C++ compilers from the 90s. And the software was probably written in the 90s, too.
[0] While a competent sysadmin expects that a failure to provide valid values in a configuration file will result in a service not functioning, our typical interaction with modern software comes with the expectation that an invalid value provided to a configuration form will result in a rejection of the value. Even in the cases of Apache/nginx, they provide a method to check your configuration before using it -- just to be safe to make sure you didn't leave out a semicolon/closing brace/
I was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (…
1. Thanks for the awesome TIL
2. Rumors fly about old versions of Windows, OS/2, etc still being actively used. I like to pin down and file away usage/year correlations, where possible. What sort of timeframe (roughly) was Win98 in active use here?
3. Regarding [1], I have an ancient [runs downstars to check] Compaq Prosignia 300 server here and I discovered in the (DR-DOS based) BIOS at one point that the serial port's electrical behavior can be customized between being edge triggered or level triggered. (Mildly interesting machine. Insists it has 83MB of RAM. Has the FOOF bug. Its SCSI disks make nice noises when they spin up.) Maybe this is related.
I wouldn’t consider that to be a technicians misstep. That’s poor software design.
Yeah, but nobody ever seems to throw the UX designers under the bus, right?
——
You are assuming the entry form was put together with the assistance of a UX designer.
In my experience, back office or configuration software rarely is seen as important enough by the Powers-That-Be to justify all the “extra research and design effort” required.
Usually work like that is tossed over to whichever mid-level software engineer has an extra cycle or two during the sprint.
This is not to imply that software engineers are always bad at UX. In fact, most engineers I’ve worked closely with care a lot about the end users’ experiences, however, when push comes to shove, their leaders (or the overseeing Product teams they are accountable to) push for rapid delivery of features to reach market parity, rather than spending the extra few days to formally validate the right design decisions were made and if the implementation of those designs were solidly understandable.
Going back to your original comment: We should, as an industry, hold designers accountable if their decisions lead to detrimental consequences, especially ones which could be anticipated, like this one.
However, we should also recognize this is never the failure of solely the designer, but rather indicators of systematic issues of the extended team.
A poorly designed feature/function which makes it way through...
* pre-dev stakeholder reviews
* development and implementation
* quality assurance
* acceptance testing
It is a failure of culture and leadership if no one along the chain had been comfortable or able to raise a flag if they disagreed or foresaw a problem.
Edit: Fixed formatting and this ended up to be longer than I expected when I stared typing. Also, typos.
This sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do
I commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely,…