So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.
[deleted]
UK air traffic control meltdown
91–100 of 459 posts
Re: UK air traffic control meltdown
#92Earlier quoted context omitted.
The problem is systems written in the 1970s in FORTRAN to run on Mainframes don't speak JSON.
Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).
Have you ever been involved in such a migration?
It’s invariably a complete clusterfuck.
Re: UK air traffic control meltdown
#93This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.
The bug here was a processing one, having the data in json would make no difference.
Re: UK air traffic control meltdown
#94Earlier quoted context omitted.
Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…
> Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed. How many KeyError exceptions have brought down your whole server? It doesn't happen because whoever coded your web framework knows better and added a big try-catch around the code which handles individual requests. That way you get a 500 error on the spec…
> big try-catch around the code which handles individual requests.
I mean, that's assuming the code isolating requests is also bug free. You just don't know.
Re: UK air traffic control meltdown
#95Earlier quoted context omitted.
Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.
I have seen a bad outages caused by valid JSON whose consumer implemented something incorrectly. I agree with dundarius that "doing this in JSON" would not have changed the likelihood the bug could have manifested.
No change at all? I find that hard to believe. There's also a data design problem here, but the structure of JSON would aid in, not subtract from, that process.
The question at hand is: "heavily structured data vs. a blob of text as input into a complex algorithm, which one is preferred?"
Unless you're lying, you'd choose the former given the option.
Re: UK air traffic control meltdown
#96Earlier quoted context omitted.
The problem is systems written in the 1970s in FORTRAN to run on Mainframes don't speak JSON.
Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).
I guess we should rewrite it in Rust.
Airplane logistics feels like one of the most complicated systems running today. A single airline has to track millions of entities: planes, parts, engineers, luggage, cargo, passengers, pilots, gate agents, maintenance schedules, etc. Most of which was created all before best-practices were a thing. Not only is the software complex, but there are probably millions of devices in the world expecting exactly format X and will never be upgraded.
I have no doubt that eventually the software will be Ship of Thesus-ed into something approaching sanity, but there are likely to be glaciers of tech debt which cannot be abstracted away in anything less than decades of work.
Re: UK air traffic control meltdown
#97This is an interesting engineering problem and I'm not sure what the best approach is. Fail safe and stop the world, or keep running and risk danger? I imagine critical systems like trading/aerospace have this worked out to some degree.
Absolutely no idea on what is correct, but I love to reference this article on software practices at NASA[0], They Write the Right Stuff.
[0] https://www.fastcompany.com/28121/they-write-right-stuff
Re: UK air traffic control meltdown
#98I wish the article contained some explanation of why the processing for NATS requires looking at both the ADEXP waypoints and the ICAO4444 waypoints (not a criticism per se, it may not have been addressed in the underlying report). Just looking at the ADEXP seems sufficient for the UK segment logic.
I'm guessing it has something to do with how ICAO4444 is technically human readable, and how in some meaningful sense, pilots and ATC staff "prefer" it. e.g., maybe all ICAO4444 waypoints are "significant" to humans (like international airports), whereas ADEXP waypoints are often "insignificant" (local airports, or even locations without any runway at all).
Of course with 20/20 hindsight, it seems obviously incorrect to loop through the ICAO4444 waypoints in their entirety, instead of "resuming" from an advanced position. But why look at them at all?
Re: UK air traffic control meltdown
#99A day I don't want to remember.
Took me 15 hours to reach my destination instead of 2.
Had to take train, bus, then train again. 30 minutes after I had booked my tickets, everything was fully booked for two days.
Re: UK air traffic control meltdown
#100Earlier quoted context omitted.
> Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed. How many KeyError exceptions have brought down your whole server? It doesn't happen because whoever coded your web framework knows better and added a big try-catch around the code which handles individual requests. That way you get a 500 error on the spec…
Crash is a feature, though. It's not like exceptions raises by itself into interpreter specifications. It's just that it so happens that Web apps ain't need no airbags that slow down businesses.
I love that phrasing, I'm gonna use that from now on when talking about low-stakes vs high-stakes systems.