So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.
Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?
UK air traffic control meltdown
171–180 of 459 posts
Re: UK air traffic control meltdown
#172Earlier quoted context omitted.
Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…
That reasoning is fine, but it rather seems that the programmers triggered this catastrophic "stop the world" error because they were not thorough enough considering all scenarios. As TA expounds, it seems that neither formal methods nor fuzzing were used, which would have gone a long way flushing out such errors.
Yes. But also, it's an ATC system. Its primary purpose "is to prevent collisions..." [1].
If the system encounters a "this should never happen!" error, the correct move is to shut it down and ground air traffic. (The error shouldn't have happened in the first place. But the shutdown should have been more graceful.)
Re: UK air traffic control meltdown
#173Earlier quoted context omitted.
Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…
And that's why I never (or very rarely) put "this should never happen" exceptions anymore in my code Because you eventually figure out that, yes, it does happen
Re: UK air traffic control meltdown
#174So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.
Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?
Re: UK air traffic control meltdown
#175Earlier quoted context omitted.
Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.
I'm no aviation safety controls expert but it seems to me that there are two types of controls that should be in place: 1. Process controls: What do we do when this breaks for any reason. 2. Engineering controls: What can we do to keep this from breaking in the first place? Both of them seem to be somewhat essential for a truly safe system.
Re: UK air traffic control meltdown
#176Re: UK air traffic control meltdown
#177Earlier quoted context omitted.
> the phenomenon has even got a name - "poison flight plan". Maybe, but it must not be a common phrase because your comment is the first result when I search for it. And it is also mentioned in this article: http://www.aero-news.net/subsite.cfm?do=main.textpost&id=ce2... And that's about it? Do you have any other sources?
I think that term was invented four days ago by that article writer. There are four other occurrances before then and they're about PS2 games.
Re: UK air traffic control meltdown
#178Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…
I wouldn't expect level 1 and level 2 to be able to diagnose a problem like this level 3 (devs) should have been brought in much quicker though
Re: UK air traffic control meltdown
#179> The programming style is very imperative Is that supposed to be a meaningful statement?
Re: UK air traffic control meltdown
#180> Human lives were kept safe at all times
> The consequence of all this was not that any human lives were put in danger, ..
When you're arguing that cancelling 2000 flights cost £100M and that no human danger was incurred, something should feel off. That might be around 600k humans who weren't able to be where they felt they needed to be. Did they have somewhere safe to sleep? Did they have all the medications they needed with them? Did they have to miss a scheduled surgery? Could we try to measure the effect on their well-being in aggregate, using a metric other than the binary state of alive or facing imminent death? You get the idea.
Of course I agree with the version of the claim that says that no direct danger was caused from the point of view of the failing-safe system. But when you're designing a system, it ought to be part of your role to wonder where risk is going as you more stringently displace it from the singular system and source of risk that you maintain.