Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

171–180 of 459 posts

Re: UK air traffic control meltdown

#171

So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

I would guess because humans have to read this and ascertain meaning from it. Not everyone is a technical resource.

Re: UK air traffic control meltdown

#172
post #155

Earlier quoted context omitted.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

That reasoning is fine, but it rather seems that the programmers triggered this catastrophic "stop the world" error because they were not thorough enough considering all scenarios. As TA expounds, it seems that neither formal methods nor fuzzing were used, which would have gone a long way flushing out such errors.

> it rather seems that the programmers triggered this catastrophic "stop the world" error because they were not thorough enough considering all scenarios

Yes. But also, it's an ATC system. Its primary purpose "is to prevent collisions..." [1].

If the system encounters a "this should never happen!" error, the correct move is to shut it down and ground air traffic. (The error shouldn't have happened in the first place. But the shutdown should have been more graceful.)

[1] https://en.wikipedia.org/wiki/Air_traffic_control

Re: UK air traffic control meltdown

#173

Earlier quoted context omitted.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

And that's why I never (or very rarely) put "this should never happen" exceptions anymore in my code Because you eventually figure out that, yes, it does happen

This here is the true takeaway. The bar for writing "this should never happen" code must be set so impossibly high that it might as well be translated into "'this should never happen' should never happen"

Re: UK air traffic control meltdown

#174

So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

The names have to be entered manually by pilots, if e.g. they change the route. They have to be transmitted over the air by humans. So they must be short ans simple.

Re: UK air traffic control meltdown

#175
post #149

Earlier quoted context omitted.

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

I'm no aviation safety controls expert but it seems to me that there are two types of controls that should be in place: 1. Process controls: What do we do when this breaks for any reason. 2. Engineering controls: What can we do to keep this from breaking in the first place? Both of them seem to be somewhat essential for a truly safe system.

It's very hard to ensure you capture every single possible failure mode. Yes, the engineering control is important but it's not the most critical. What to do if it does fail (for any reason) is the truly critical control, because it solves for the possibility of not knowing every possible way something might fail and therefore missing some way to prevent a failure

Re: UK air traffic control meltdown

#177
post #152
post #70

Earlier quoted context omitted.

> the phenomenon has even got a name - "poison flight plan". Maybe, but it must not be a common phrase because your comment is the first result when I search for it. And it is also mentioned in this article: http://www.aero-news.net/subsite.cfm?do=main.textpost&id=ce2... And that's about it? Do you have any other sources?

I think that term was invented four days ago by that article writer. There are four other occurrances before then and they're about PS2 games.

This term was in wide circulation when I was consulting at NATS in the 2000-2005 time frame.

Re: UK air traffic control meltdown

#178
post #30
post #19

Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…

I wouldn't expect level 1 and level 2 to be able to diagnose a problem like this level 3 (devs) should have been brought in much quicker though

Having worked in tech support: level 3 (Devs) should have described their source code structure to level 2, and let them access it when they needed it.

Re: UK air traffic control meltdown

#179

> The programming style is very imperative Is that supposed to be a meaningful statement?

Yes, typically it would be used to mean things like the code mutates data in place rather than using persistent data structures, explicitly loops over data rather than using higher-order map, fold etc. operations, and explicitly checks tag bits rather than using sum types.

Re: UK air traffic control meltdown

#180
Great post. This part goes too far, I think:

> Human lives were kept safe at all times

> The consequence of all this was not that any human lives were put in danger, ..

When you're arguing that cancelling 2000 flights cost £100M and that no human danger was incurred, something should feel off. That might be around 600k humans who weren't able to be where they felt they needed to be. Did they have somewhere safe to sleep? Did they have all the medications they needed with them? Did they have to miss a scheduled surgery? Could we try to measure the effect on their well-being in aggregate, using a metric other than the binary state of alive or facing imminent death? You get the idea.

Of course I agree with the version of the claim that says that no direct danger was caused from the point of view of the failing-safe system. But when you're designing a system, it ought to be part of your role to wonder where risk is going as you more stringently displace it from the singular system and source of risk that you maintain.

Post reply on HN