Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

131–140 of 459 posts

Re: UK air traffic control meltdown

#131

Post by adrianmonk »

asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error.

If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have continued. Or maybe you're getting the error because the software has a catastrophic new bug that will silently corrupt all the other flight plans and get people killed. You don't know whether it is or isn't safe to continue, so you stop.

Re: UK air traffic control meltdown

#132

Post by lgeorget »

jacquesm wrote: post #126

Trusted input rarely should be trusted. It's input. You need to validate it as if it is hostile and have a process for dealing with malformed input. Now of course, standing by the sidelines it is easy to criticize and I'm sure whoever worked on this wasn't stupid. But I've seen this error often enough now in practice that I think that it needs to be drilled into programmers heads more forcefully: stuff is only valid…

From what I gathered from the article, the input WAS valid. It's the software that was unable to handle a specific case of valid input.

Re: UK air traffic control meltdown

#133

Post by ummonk »

Spivak wrote: post #32
asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

That it's safety critical is all the more reason it should fail gracefully (albeit surfacing errors to warn the user). A single bad flight plan shouldn't jeopardize things by making data on all the other flight plans unavailable.

Re: UK air traffic control meltdown

#134

Post by amiga386 »

darkclouds wrote: post #7

Interesting to see that flight plans over the UK have to be filed 4 hours in advance. No mention of plane, pilot, passenger and cargo manifests. So why the 4 hour lead time, is this the time it takes UK Authorities to look people up or workout if the cargo could be dangerous in an airborne Anthrax (Gruinard) Island [1] or Japanese subway Sarin [2], or an IRA favourite, fertilizer bomb thats bypassed the usual purchas…

> so why the 4 hour lead time

To answer your question without conspiracy drivel, let's look up CAP 694: The UK Flight Planning Guide [0]

Chapter 1

> 6.1 The general ICAO requirement is that FPLs should be filed on the ground at least 60 minutes before clearance to start-up or taxi is requested. The "Estimated Off Block Time" (EOBT) is used as the planned departure time in flight planning, not the planned airborne time.

> 6.3 IFR flights on the North Atlantic and on routes subject to Air Traffic Flow Management, should be filed a minimum of 3 hours before EOBT (see Chapter 4).

Chapter 4

> 1.1 The UK is a participating State in the Integrated Initial Flight Plan Processing System (IFPS), which is an integral part of the Eurocontrol centralised Air Traffic Flow Management (ATFM) system.

> 4.1 FPLs should be filed a minimum of 3 hours before Estimated Off Block Time (EOBT) for North Atlantic flights and those subject to ATFM measures, and a minimum of 60 minutes before EOBT for all other flights.

So the answer is because the UK is part of a Europe-wide air traffic control system, which hands out full flight plans to all the relevant authorities for each airspace, and they decided 3 hours is needed so that all possible participants can get their shit together and tell you if they accept the plan or not.

An entirely separate system exists to share Advanced Passenger Information, i.e. passenger manifests [1], and it goes even further that airlines share your overall identity with each other, known as a Passenger Name Record [2], and a variety of countries, led by the USA, insist on this information in advance before the plane is allowed to take off [3]

If you're going to be paranoid, please work with known facts instead of speculating.

[0] https://publicapps.caa.co.uk/docs/33/CAP%20694.pdf

[1] https://en.wikipedia.org/wiki/Advance_Passenger_Information_...

[2] https://en.wikipedia.org/wiki/Passenger_name_record

[3] https://en.wikipedia.org/wiki/United_States%E2%80%93European...

Re: UK air traffic control meltdown

#135

Post by mannykannot »

RyJones wrote: post #128
macguillicuddy wrote: post #69

Earlier quoted context omitted.

For the most part (although there are important exceptions), IFR flights are always in radar contact with a controller. The flight plan is tool allows ATC and the plane to agree a route so that they don't have to be constantly communicating. ATC 'clears' a plane to continue on the route to a given limit, and expects the plane to continue on the plan until that limit unless they give any future instructions. In this r…

Mostly yes; however, there are large parts of the Atlantic and Pacific where that isn't true (radar contact). I know the Atlantic routes are frequently full of plans that left the US and Canada heading to the UK. I have no idea what percent of the volume into the UK comes from outside radar control; if they asked a flight to divert, that may open multiple other cans of worms.

> If they asked a flight to divert, that may open multiple other cans of worms.

Any ATC system has to be resilient enough to handle a diversion on account of things like bad weather, mechanical failure or a medical emergency. In fact, I would think the diversion of one aircraft would be less of a problem than those caused by bad weather, and certainly less than the problem caused by this failure. Furthermore, I would guess that the mitigation would be just to manually direct the flight according to the accepted flight plan, as it was a completely valid one.

One of the many problems here is that they could not identify the problem-triggering flight plan for hours, and only with the assistance of the vendor's engineers. Another is that the system had immediately foreclosed on that option anyway, by shutting down.

Re: UK air traffic control meltdown

#136

Post by sergers »

vixen99 wrote: post #54
sergers wrote: post #16

Earlier quoted context omitted.

Well that's DailyMail for you, where they tag anything parenting or healthy as "femail" section... cause you know only women are looking at that stuff. Lol. Anyways I actually think that's just reasonable response, system goes down/related system goes down , and in reviewing they are making frivolous updates to names that aren't needed. I would question these updates (while they may be minor part of overall updates o…

Is this the UK or US edition? It's always easy fun to have a go at the Daily Mail which presumably you read regularly else you wouldn't be commenting. Its sin seems to be that it's not a serious broadsheet. It's a tabloid with very broad appeal that has to be profitable and therefore tries to reflect the requirements of the British public for such a publication. Perhaps you should lower your expectations. 'Tag anythi…

Dailymail is actually site a frequent multiple times a day everyday.

not all content is for everyone, but they got something, they are definitely a tabloid style.

they narrate particular views to the public but cover all different contents, and alot of content i would consider advertisements/plug than actual articles.

i would guess a highly elderly/conservative majoroity base

they pander to lowest common denominator, which is fine -- they are a for profit news/tabloid, i find some of it entertaining (As per daily visits).

do you work for them/just a big fan for doing all that digging in defense of DM overexaggerating i made that ALL content like that is in that category? i didnt take my own comment all that seriously so honest ask.

Re: UK air traffic control meltdown

#137

Post by jacquesm »

lgeorget wrote: post #132
jacquesm wrote: post #126

Trusted input rarely should be trusted. It's input. You need to validate it as if it is hostile and have a process for dealing with malformed input. Now of course, standing by the sidelines it is easy to criticize and I'm sure whoever worked on this wasn't stupid. But I've seen this error often enough now in practice that I think that it needs to be drilled into programmers heads more forcefully: stuff is only valid…

From what I gathered from the article, the input WAS valid. It's the software that was unable to handle a specific case of valid input.

That's fine, and is exactly the kind of case that I was thinking of: your software has a different idea of what is valid than an upstream piece of software, so from your perspective it is invalid. So you need to pull this message out of the stream, sideline it so it can be looked at by someone qualified enough to make the call of what's the case (because it could well be either way) and processing for all other messages should continue as normal. After all the only reason you can say with confidence that it in fact was valid is because someone looked at it! You can only do that well after the fact.

A message switch [1] that I worked on had to deal with messages sources from 100's of different parties and while in principle everybody was working from the same spec (CCITT [2]) every day some malformed messages would land in the 'error' queue. Usually the problem was on the side of the sender, but sometimes (fortunately rarely) it wasn't and then the software would be improved to be able to handle that case correctly as well. Given the size of the specs and the many variations on the protocols it wasn't weird at all to see parties get confused. What's surprising is that it happens as rarely as it does.

The big takeaway here should be that even if something happens very rarely it should still not result in a massive cascade, the system should handle this gracefully.

[1] https://www.kvsa.nl/en/

[2] https://en.wikipedia.org/wiki/Group_4_compression

Re: UK air traffic control meltdown

#138

Post by onetimeuse92304 »

jacquesm wrote: post #126

Trusted input rarely should be trusted. It's input. You need to validate it as if it is hostile and have a process for dealing with malformed input. Now of course, standing by the sidelines it is easy to criticize and I'm sure whoever worked on this wasn't stupid. But I've seen this error often enough now in practice that I think that it needs to be drilled into programmers heads more forcefully: stuff is only valid…

This really isn't about input. Whether it comes from outside or produce inside the application, the reality is that everything can have bugs. A correct input can cause a buggy application to fail. So while verifying input is obviously an important step, it is not even a beginning if you are really looking to building reliable software.

What really is the heart of the matter is for the entire thing to be allowed to crash due to a problem with single transaction.

What you really want to do is to have firewalls. For example, you want a separate module that runs individual transactions and a separate shell that orchestrates everything but has no or very limited contact with the individual transactions. As bad as giving up on processing a single aircraft is, allowing the problem to cascade to entire system is way worse.

What's even more tragic about this monumental waste of resources is that the knowledge about how to do all of this is readily available. The aerospace and automotive industry have very high development standards along with people you can hire who know those standards and how to use them to write reliable software.

Re: UK air traffic control meltdown

#139

Post by failbuffer »

> The manufacturer was able to offer further expertise including analysis of lower-level software logs which led to identification of the likely flight plan that had caused the software exception.

This part stood out to me. I've found it super helpful to include a reference to which piece of days in working with in log messages and exceptions. It helps isolated problems so much faster.

Re: UK air traffic control meltdown

#140

Post by seabass-labrax »

SoftTalker wrote: post #111
throw7 wrote: post #62

Well, I certainly hope they've at least stopped issuing waypoints with identical names... although it wouldn't surprise me if geographically-distant is the best we can do as a species.

They appear to be sequences of 5 upper-case letters. Assuming the 26-character alphabet, that should allow for nearly 12 million unique waypoint IDs. The world is a big place but that seems like it should be enough. The more likely problem is that there is (or was) no internationally-recognized authority in charge of handing out waypoint IDs, so we have at least legacy duplicates if not potential new ones.

You have to reduce that to the (still massive) set of IDs that are somewhat pronounceable in languages that use the Latin script. You don't want to be the air traffic controller trying to work out how to say 'Lufthansa 451, fly direct QXKCD'. Nonetheless, I think the there is little cause for concern about changing existing IDs. There might be sentimental attachment, but it takes barely a few flights before the new IDs start sticking, and it's not like pilots never fly new routes.
Post reply on HN