Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

301–310 of 459 posts

Re: UK air traffic control meltdown

#301

Bugs happen. Fact of being written by fleshy meatballs. What should also have been highlighted is that they clearly had no easy way of finding the specific buggy input in the logs nor simulating it without contacting the manufacturer.

It sounds like a simple functional smoke test throwing random flight plans at the system would have eventually (and probably pretty soon) triggered this. I hope they at least do it now.

This reminds me of: https://danluu.com/wat/

Re: UK air traffic control meltdown

#302

I wish the article contained some explanation of why the processing for NATS requires looking at both the ADEXP waypoints and the ICAO4444 waypoints (not a criticism per se, it may not have been addressed in the underlying report). Just looking at the ADEXP seems sufficient for the UK segment logic. I'm guessing it has something to do with how ICAO4444 is technically human readable, and how in some meaningful sense,…

They use the ADEXP to determine which part of the route is in the UK. Because the auto generated points are ATC area handover points. So this data is the best way so see which part of the route is within the UK airspace.

Then it needs to find the ICAO part that corresponds, because the controller needs to use the ICAO plan that the pilot has.

If the controller sees other (auto generated) waypoints that the pilots don't have you get problems during operation. A simple example is that controllers can tell pilots to fly in a straight line to a specific point on their filed route (and do so quite often). The pilot is expected to continue the filed route from that point onwards.

They can also tell a pilot to fly direct to some random other point (this also happens but less often). The pilot is then not expected to pick up a route after that point.

The radio instruction for both is exactly the same, the only difference is whether the point is part of the planned route or not. So the controller needs to see the exact same route as the pilots have, not one with additional waypoints added by the IFPS system.

Re: UK air traffic control meltdown

#303

Earlier quoted context omitted.

I agree with the general sentiment "if you see an unexpected error, STOP", but I don't really think that applies here. That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processi…

Except that you can't be sure this bad flight plan doesn't contain information that will lead to a collision. The system needs to maintain the integrity of all plans it sees. If it can't process one, and there's the risk of a plane entering airspace with a bad flight plan, you need to stop operations.

>> Except that you can't be sure this bad flight plan doesn't contain information that will lead to a collision.

Flight plans don't contain any information relevant for collision avoidance. They only say when and where the plane is expected to be. There is not enough specificity to ensure no collisions. Things change all the time, from late departures, to diverting around bad weather. On 9/11 they didn't have every plane in the sky file a new flight plan carefully checked against every other...

Re: UK air traffic control meltdown

#304
post #190

Great post. This part goes too far, I think: > Human lives were kept safe at all times > The consequence of all this was not that any human lives were put in danger, .. When you're arguing that cancelling 2000 flights cost £100M and that no human danger was incurred, something should feel off. That might be around 600k humans who weren't able to be where they felt they needed to be. Did they have somewhere safe to sl…

I mean it could have also saved lives by that logic. Did someone missing their flight mean they also missed a terrible pileup on the roadways after landing? We can imagine pretty much any scenario here.

But how many lives were saved by the reduced carbon emissions that were not produced by the cancelled flights?

Re: UK air traffic control meltdown

#305
post #76

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same reason. It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do. The bit…

No validation, anddd this point from the article stood out to me: --- The programming style is very imperative. Furthermore, the description sounds like the procedure is working directly on the textual representation of the flight plan, rather than a data structure parsed from the text file. This would be quite worrying, but it might also just be how it is explained. --- Given that description, I'd be surprised if it wasn't just running a regex / substring matches against the text and there's no classes / objects / data structure involved. Bearing in mind this is likely decades old C code that can't be rewritten or replaced because the entirety of the UK's aviation runs on it.

Re: UK air traffic control meltdown

#306
post #32

Earlier quoted context omitted.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

That's like saying that because one browser tab tried to parse some invalid JSON then my whole browser should crash.

Nonsense comparison, your browser's tabs are de facto insulated from each other, flight paths for 7000 daily planes over the UK literally share the same space.

Re: UK air traffic control meltdown

#307
post #66

Earlier quoted context omitted.

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it. The thing that blows my mind is that this was apparently the…

One in a 15M chance with 7000 daily flies over the UK handled by nats meant it had a probability to happen at least once in 69 months, it took few months less.

Re: UK air traffic control meltdown

#309

I had been considering becoming an air traffic controller myself, and it rather tickles me to think I might have missed my once-in-a-lifetime opportunity to direct aircraft with the original pen-and-paper flight strip mechanism in the 21st century! Completely safe, excruciatingly low-capacity, and sounds like awfully good fun as a novelty (for the willing ATC, not the passengers stuck on the ground, I hasten to add).

Quite few non major airports are still heavily pen and paper reliant methods to some degree.

An example are islands that serve few flights per week and can't justify heavy update investments.

Airplanes are generally spaced by hours and you need to do your math about where the airplanes are by hand. But again there's so little planes that risks are minimal.

Re: UK air traffic control meltdown

#310
post #179

> The programming style is very imperative Is that supposed to be a meaningful statement?

Yes, typically it would be used to mean things like the code mutates data in place rather than using persistent data structures, explicitly loops over data rather than using higher-order map, fold etc. operations, and explicitly checks tag bits rather than using sum types.

Fine, I'll give you that (sounds like a generic description) but there's nothing like that from the description given in the article and the paragraph immediately before that statement. It's almost as if the author completely made that up.
Post reply on HN