Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

111–120 of 459 posts

Re: UK air traffic control meltdown

#111

Post by SoftTalker »

throw7 wrote: post #62

Well, I certainly hope they've at least stopped issuing waypoints with identical names... although it wouldn't surprise me if geographically-distant is the best we can do as a species.

They appear to be sequences of 5 upper-case letters. Assuming the 26-character alphabet, that should allow for nearly 12 million unique waypoint IDs. The world is a big place but that seems like it should be enough. The more likely problem is that there is (or was) no internationally-recognized authority in charge of handing out waypoint IDs, so we have at least legacy duplicates if not potential new ones.

Re: UK air traffic control meltdown

#112

Post by P-Nuts »

blibble wrote: post #30
Diggsey wrote: post #19

Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…

I wouldn't expect level 1 and level 2 to be able to diagnose a problem like this level 3 (devs) should have been brought in much quicker though

You don’t need a complete diagnosis if you can spit out enough debug info that says, “oops shat the bed while working with this flight plan”, then the support people can remove the one that’s causing you to fail, restart the system, and tell ATC to route that one manually.

Re: UK air traffic control meltdown

#113

Post by dundarious »

masklinn wrote: post #109
dundarious wrote: post #98

I wish the article contained some explanation of why the processing for NATS requires looking at both the ADEXP waypoints and the ICAO4444 waypoints (not a criticism per se, it may not have been addressed in the underlying report). Just looking at the ADEXP seems sufficient for the UK segment logic. I'm guessing it has something to do with how ICAO4444 is technically human readable, and how in some meaningful sense,…

Possibly it needs the ICAO information to communicate with some systems, but has to work in ADEXP to have sufficient granularity (the essay mentions the possibility of “clipping”, a flight going through the UK between two ICAO waypoints).

Yes, I'm essentially wanting to know more about those existing ICAO-based systems, be they machine or not.

Re: UK air traffic control meltdown

#114

Post by FeepingCreature »

SillyUsername wrote: post #80

Bugs happen. Fact of being written by fleshy meatballs. What should also have been highlighted is that they clearly had no easy way of finding the specific buggy input in the logs nor simulating it without contacting the manufacturer.

No way or no procedure.

Re: UK air traffic control meltdown

#115

Post by FeepingCreature »

passwordoops wrote: post #46
redleader55 wrote: post #42

I imagine, for this kind of system, there is only one supplier. Why not force that supplier, as part of their 10-15 yr contract, to publish the source code for everything, not necessarily as FOSS. This way if there are bugs they can be reported and fixed.

I agree. But this would assume that: 1- the people writing and approving the specs even understand why this might be a good suggestion 2- the people ultimately approving the contract aren't in bed with the supplier

3- the people operating the system are capable of maintaining its source code

Re: UK air traffic control meltdown

#116

Post by marcosdumay »

numpad0 wrote: post #86
krisoft wrote: post #58

Earlier quoted context omitted.

> Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed. How many KeyError exceptions have brought down your whole server? It doesn't happen because whoever coded your web framework knows better and added a big try-catch around the code which handles individual requests. That way you get a 500 error on the spec…

Crash is a feature, though. It's not like exceptions raises by itself into interpreter specifications. It's just that it so happens that Web apps ain't need no airbags that slow down businesses.

On a multi-user system, only partial crashes are features. Total crashes are bugs.

A web server is a multi-user system, just like a country's air traffic control.

Re: UK air traffic control meltdown

#117

Post by madeofpalk »

Spivak wrote: post #102
madeofpalk wrote: post #75

Earlier quoted context omitted.

That's like saying that because one browser tab tried to parse some invalid JSON then my whole browser should crash.

Well yes because you're describing a system where there are really low stakes and crash recovery is always possible because you can just throw away all your local state. The flip side would be like a database failing to parse some part of its WAL log due to disk corruption and just said, "eh just delete those sections and move on."

Crash the tab and allow all the others to carry on!

The problem here is that one individual document failed to parse.

Re: UK air traffic control meltdown

#118

Post by vb-8448 »

rglover wrote: post #106
vb-8448 wrote: post #74

Earlier quoted context omitted.

It won't fix anything. JSON is the "standard" today, 15 years ago it was XML and in 15 years we will have protobuf or another new standard.

Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.

> codify migration practices

I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating. The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.

Re: UK air traffic control meltdown

#119

Post by jandrese »

slt2021 wrote: post #66
cratermoon wrote: post #5

Earlier quoted context omitted.

> why could the system not put the failed flight plan in a queue Because it doesn't look at the data as a "flight plan" consisting of "way points" with "segments" along a "route" that has any internal self-consistency. It's a bag of strings and numbers that's parsed and the result passed along, if parsing is successful. If not, give up. In this case fail the entire system and take it out of production. Airline indust…

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it.

The thing that blows my mind is that this was apparently the first time this situation had happened after 15 million records processed. I would have expected it to trigger much more often. It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.

Re: UK air traffic control meltdown

#120

Post by mprovost »

rglover wrote: post #72
tristor wrote: post #59

Earlier quoted context omitted.

The problem is systems written in the 1970s in FORTRAN to run on Mainframes don't speak JSON.

Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).

No migration of this magnitude is blocked because of engineers not "bothering" to start the process. Imagine how many approvals you'd need, plus getting budget from who-knows how many government departments. Someone is paying for your time as an engineer and they decide what you work on. I'm glad we live in a world where engineers can't just decide to rewrite a life or death system because it's written in an old(er) programming language. (Not that there is any evidence that this specific system is written in anything older than C++ or maybe Ada.)
Post reply on HN