Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

101–110 of 459 posts

Re: UK air traffic control meltdown

#101

Post by numpad0 »

rglover wrote: post #68
dundarious wrote: post #65

Earlier quoted context omitted.

Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.

Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.

To me and XML-ified this would look more nightmarish than the status quo... it's just brief, space separated and \n terminated ASCII. No need to overcomplicate things this simple.

Re: UK air traffic control meltdown

#102

Post by Spivak »

madeofpalk wrote: post #75
Spivak wrote: post #32

Earlier quoted context omitted.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

That's like saying that because one browser tab tried to parse some invalid JSON then my whole browser should crash.

Well yes because you're describing a system where there are really low stakes and crash recovery is always possible because you can just throw away all your local state.

The flip side would be like a database failing to parse some part of its WAL log due to disk corruption and just said, "eh just delete those sections and move on."

Re: UK air traffic control meltdown

#103

Post by zimpenfish »

madeofpalk wrote: post #75
Spivak wrote: post #32

Earlier quoted context omitted.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

That's like saying that because one browser tab tried to parse some invalid JSON then my whole browser should crash.

No, it's more like saying your browser has detected possible internal corruption with, say, its history or cookies database and should stop writing to it immediately. Which probably means it has to stop working.

Re: UK air traffic control meltdown

#104

Post by rglover »

fullspectrumdev wrote: post #92
rglover wrote: post #72

Earlier quoted context omitted.

Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).

Have you ever been involved in such a migration? It’s invariably a complete clusterfuck.

I haven't, but I'd love to. My approach wouldn't be very "HR friendly," though.

Re: UK air traffic control meltdown

#105

Post by dundarious »

rglover wrote: post #95
schainks wrote: post #88

Earlier quoted context omitted.

I have seen a bad outages caused by valid JSON whose consumer implemented something incorrectly. I agree with dundarius that "doing this in JSON" would not have changed the likelihood the bug could have manifested.

No change at all? I find that hard to believe. There's also a data design problem here, but the structure of JSON would aid in, not subtract from, that process. The question at hand is: "heavily structured data vs. a blob of text as input into a complex algorithm, which one is preferred?" Unless you're lying, you'd choose the former given the option.

The issue is using both ADEXP and ICAO4444 waypoints, and doing so in a sloppy way. For the waypoint lists, there is no issue with structurelessness -- the fact that they're lists is pretty obvious, even in the existing formats. Adding some ["",] would not have helped the specific problem, as the relevant structure was already perfectly clear to the implementers. I am not lying when I say the bug would have been equally likely in a JSON format in this specific case.

Re: UK air traffic control meltdown

#106

Post by rglover »

vb-8448 wrote: post #74
rglover wrote: post #31

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.

It won't fix anything. JSON is the "standard" today, 15 years ago it was XML and in 15 years we will have protobuf or another new standard.

Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.

Re: UK air traffic control meltdown

#107

Post by SoftTalker »

piva00 wrote: post #82
Spivak wrote: post #32

Earlier quoted context omitted.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

> Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. What surprised me more is that the amount of data existing for all waypoints on the globe is quite small, if I were to implement a feature that query by their names as an identifier the first thing I'd do…

If there are duplicate waypoint IDs, they are not close together. They can be easily eliminated by selecting the one that is one hop away from the prior waypoint. Just traversing the graph of waypoints in order would filter out any unreachable duplicates.

Re: UK air traffic control meltdown

#108

Post by zimpenfish »

rglover wrote: post #68
dundarious wrote: post #65

Earlier quoted context omitted.

Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.

Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.

> The algorithm was buggy because the input data is a nightmare.

No, the algorithm was "buggy" because it didn't account for the entry to and exit points from the UK to have the same designation because they're supposed to be geographically distant (they were 4000Nm apart!) and the UK ain't that big.

Re: UK air traffic control meltdown

#109

Post by masklinn »

dundarious wrote: post #98

I wish the article contained some explanation of why the processing for NATS requires looking at both the ADEXP waypoints and the ICAO4444 waypoints (not a criticism per se, it may not have been addressed in the underlying report). Just looking at the ADEXP seems sufficient for the UK segment logic. I'm guessing it has something to do with how ICAO4444 is technically human readable, and how in some meaningful sense,…

Possibly it needs the ICAO information to communicate with some systems, but has to work in ADEXP to have sufficient granularity (the essay mentions the possibility of “clipping”, a flight going through the UK between two ICAO waypoints).

Re: UK air traffic control meltdown

#110

Post by supernova87a »

Did the creator of the flight plan software engage in adversarial testing to see if they could break the system with badly formed flight plans? Or was / is the typical practice to mostly just see if the system meets just the "well-behaved" flight plan processing requirements? (with unit tests, etc)
Post reply on HN