UK air traffic control meltdown • Hacker News Board

Re: UK air traffic control meltdown

#51

Post by hn_throwaway_99 » Mon, Sep 11, 2023, 4:31 PM UTC

asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

To be fair that is exactly what the article said was a major problem, and which the postmortem also said was a major problem. I agree I think this is the most important issue:

> The FPRSA-R system has bad failure modes

> All systems can malfunction, so the important thing is that they malfunction in a good way and that those responsible are prepared for malfunctions.

> A single flight plan caused a problem, and the entire FPRSA-R system crashed, which means no flight plans are being processed at all. If there is a problem with a single flight plan, it should be moved to a separate slower queue, for manual processing by humans. NATS acknowledges this in their "actions already undertaken or in progress":

>> The addition of specific message filters into the data flow between IFPS and FPRSA-R to filter out any flight plans that fit the conditions that caused the incident.

Re: UK air traffic control meltdown

#52

Post by dang » Mon, Sep 11, 2023, 4:32 PM UTC

Related. Others?

Coincidentally-identical waypoint names foxed UK air traffic control system - https://news.ycombinator.com/item?id=37430384 - Sept 2023 (64 comments)

UK air traffic control outage caused by bad data in flight plan - https://news.ycombinator.com/item?id=37402766 - Sept 2023 (20 comments)

NATS report into air traffic control incident details root cause and solution - https://news.ycombinator.com/item?id=37401864 - Sept 2023 (19 comments)

UK Air traffic control network crash - https://news.ycombinator.com/item?id=37292406 - Aug 2023 (23 comments)

Re: UK air traffic control meltdown

#53

Post by omginternets » Mon, Sep 11, 2023, 4:32 PM UTC

laputan_machine wrote: post #33

omginternets wrote: post #29

Earlier quoted context omitted.

It’s all good. We still take cheap-shots at English food and English women ;) Edit: I lived in London for 3 years. I miss it every day.

I wouldn't worry about it, we take cheap shots at French food and French people too ;)

Sir, those are dueling words.

Re: UK air traffic control meltdown

#54

Post by vixen99 » Mon, Sep 11, 2023, 4:32 PM UTC

sergers wrote: post #16

gumballindie wrote: post #11

Earlier quoted context omitted.

[flagged]

Well that's DailyMail for you, where they tag anything parenting or healthy as "femail" section... cause you know only women are looking at that stuff. Lol. Anyways I actually think that's just reasonable response, system goes down/related system goes down , and in reviewing they are making frivolous updates to names that aren't needed. I would question these updates (while they may be minor part of overall updates o…

Is this the UK or US edition? It's always easy fun to have a go at the Daily Mail which presumably you read regularly else you wouldn't be commenting. Its sin seems to be that it's not a serious broadsheet. It's a tabloid with very broad appeal that has to be profitable and therefore tries to reflect the requirements of the British public for such a publication. Perhaps you should lower your expectations.

'Tag anything parenting or healthy ...'? No, that's not correct. Here are a few health & food related items back to mid-September that did not appear in 'female'. You are right about parenting; most parenting in the UK is still undertaken primarily (in terms of executive action) by females so items on this topic are reasonably included in 'female'. The growing number of people who don't have children probably appreciate this sub-grouping by the Mail. You may not approve but this is what happens. Single males with dependent children are not known for objecting to checking out that section. It's not forbidden.

Re: UK air traffic control meltdown

#55

Post by louthy » Mon, Sep 11, 2023, 4:33 PM UTC

sp0ck wrote: post #49

louthy wrote: post #43

Earlier quoted context omitted.

Maybe he doesn’t care if people stop reading and he’d prefer to use the language he’s most comfortable with? It’s his blog after all, not yours. Additionally, perhaps he’s making the point that a language with an expressive type system makes solving problems like this trivial.

If you don't care about readers reading it or not then what is the point to publish an article ?

I read it. Probably lots of other people did too. Presumably the people who don’t think computer science begins and ends with JavaScript

Re: UK air traffic control meltdown

#56

Post by NikolaNovak » Mon, Sep 11, 2023, 4:33 PM UTC

Diggsey wrote: post #19

Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…

Unfortunately, I work on a reasonably modern ERP system which has been customized significantly for the client and also works with wider range of client-specific data combinations that the vendor has seemingly not anticipated / other clients do not have.

What it means is that on a regular basis, teams will be woken up at 2am because a batch process aborted on bad data; AND it doesn't tell you what data / where in the process it aborted.

The only possibility is to rerun the process with crippling traces, and then manually review the logs to find the issue, remove it, and then re-run the program again (hopefully remembering to remove the trace:).

Even when all goes per plan, this can at times take more than 4 hrs.

Now, we are not running a mission-critical real-time system like air traffic; and I'm in NO way saying any of this is good; but, it may not be the case that "two level of support teams didn't know anything" - the system could just be so poorly designed that with best operational experience and knowledge, it still took that long :-On HN, we take certain level of modernity, logging, failure states, messaging, and restartability for granted; which may not be even remotely present on more niche or legacy system (again, NOT saying that's good; just indicating issue may be less with operational competence vs design). It's easy to judge from our external perspective, but we have no idea what was presented / available to support teams, and what their mandatory process is.

Re: UK air traffic control meltdown

#57

Post by joncrocks » Mon, Sep 11, 2023, 4:34 PM UTC

krisoft wrote: post #50

pimterry wrote: post #12

Earlier quoted context omitted.

To be fair, the article suggests early on that sometimes these plans are being processed for flights already in the air (although at least 4 hours away from the UK). If you can stop the specific problematic plane taking off then keeping the system running is fine, but once you have a flight in the air it's a different game. It's not totally unreasonable to say "we have an aircraft en route to enter UK airspace and we…

> "we have an aircraft en route to enter UK airspace and we don't know when or where - stop planning more flights until we know where that plane is". Flight plans don't tell where the plane is. Where is this assumption coming from?

Presumably you need to know where upcoming flights are going to be in the future (based on the plan), before they hit radar etc.

Re: UK air traffic control meltdown

#58

Post by krisoft » Mon, Sep 11, 2023, 4:37 PM UTC

Spivak wrote: post #32

asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

> Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed.

How many KeyError exceptions have brought down your whole server? It doesn't happen because whoever coded your web framework knows better and added a big try-catch around the code which handles individual requests. That way you get a 500 error on the specific request instead of a complete shutdown every time a developer made a mistake.

Re: UK air traffic control meltdown

#59

Post by tristor » Mon, Sep 11, 2023, 4:37 PM UTC

rglover wrote: post #31

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.

The problem is systems written in the 1970s in FORTRAN to run on Mainframes don't speak JSON.

Re: UK air traffic control meltdown

#60

Post by c7DJTLrn » Mon, Sep 11, 2023, 4:38 PM UTC

This is an interesting engineering problem and I'm not sure what the best approach is. Fail safe and stop the world, or keep running and risk danger? I imagine critical systems like trading/aerospace have this worked out to some degree.