Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

61–70 of 459 posts

Re: UK air traffic control meltdown

#61

Post by kccqzy »

Spivak wrote: post #32
asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

I agree with your first paragraph but your second paragraph is quite defeatist. I was involved in a quite few of "premortem" meetings where people think of increasing improbable failure modes and devise strategies for them. It's a useful meeting before larges changes to critical systems are made live. In my opinion, this should totally be a known error.

> Having found an entry and exit point, with the latter being the duplicate and therefore geographically incorrect, the software could not extract a valid UK portion of flight plan between these two points.

It doesn't take much imagination to surmise that perhaps real world data is broken and sometimes you are handed data that doesn't have a valid UK portion of flight plan. Bugs can happen, yes, such as in this case where a valid flight plan was misinterpreted to be invalid, but gracefully dealing with the invalid plan should be a requirement.

Re: UK air traffic control meltdown

#63

Post by fbdab103 »

rglover wrote: post #31

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.

Airport software predates basically every standard on the planet. I would not be surprised to learn that they have their own bizarro world implementation of ASCII, unix epoch time, etc.

Re: UK air traffic control meltdown

#64

Post by a_wild_dandan »

dang wrote: post #52

Related. Others? Coincidentally-identical waypoint names foxed UK air traffic control system - https://news.ycombinator.com/item?id=37430384 - Sept 2023 (64 comments) UK air traffic control outage caused by bad data in flight plan - https://news.ycombinator.com/item?id=37402766 - Sept 2023 (20 comments) NATS report into air traffic control incident details root cause and solution - https://news.ycombinator.com/item?i…

The recent episode of The Daily about the (US) aviation industry has convinced me that we’ll see a catastrophic headline soon. Things can’t go on like this.

Re: UK air traffic control meltdown

#65

Post by dundarious »

rglover wrote: post #31

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.

Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.

Re: UK air traffic control meltdown

#66

Post by slt2021 »

cratermoon wrote: post #5
asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

> why could the system not put the failed flight plan in a queue Because it doesn't look at the data as a "flight plan" consisting of "way points" with "segments" along a "route" that has any internal self-consistency. It's a bag of strings and numbers that's parsed and the result passed along, if parsing is successful. If not, give up. In this case fail the entire system and take it out of production. Airline indust…

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest.

seems like poor engineering

Re: UK air traffic control meltdown

#68

Post by rglover »

dundarious wrote: post #65
rglover wrote: post #31

This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.

Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.

Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.

Re: UK air traffic control meltdown

#69

Post by macguillicuddy »

joncrocks wrote: post #57
krisoft wrote: post #50

Earlier quoted context omitted.

> "we have an aircraft en route to enter UK airspace and we don't know when or where - stop planning more flights until we know where that plane is". Flight plans don't tell where the plane is. Where is this assumption coming from?

Presumably you need to know where upcoming flights are going to be in the future (based on the plan), before they hit radar etc.

For the most part (although there are important exceptions), IFR flights are always in radar contact with a controller. The flight plan is tool allows ATC and the plane to agree a route so that they don't have to be constantly communicating. ATC 'clears' a plane to continue on the route to a given limit, and expects the plane to continue on the plan until that limit unless they give any future instructions.

In this regard UK ATC can choose to do anything they like with a plane when it comes under their control - if they don't consider the flight plan to be valid or safe they can just instruct the plane to hold/divert/land etc.

I'm not sure the NATS system that failed has the ability to reject a given flight plan back upstream.

Re: UK air traffic control meltdown

#70

Post by krisoft »

NovemberWhiskey wrote: post #10

This is not the first time this has happened; the phenomenon has even got a name - "poison flight plan".

> the phenomenon has even got a name - "poison flight plan".

Maybe, but it must not be a common phrase because your comment is the first result when I search for it.

And it is also mentioned in this article: http://www.aero-news.net/subsite.cfm?do=main.textpost&id=ce2...

And that's about it? Do you have any other sources?

Post reply on HN