Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

flightglobal.com

Post reply on HN
Print view View original

81–90 of 311 posts

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#81

Post by aftbit »

wyldfire wrote: post #43
FateOfNations wrote: post #3

Good news: the system successfully detected an error and didn't send bad data to air traffic controllers. Bad News: the system can't recover from an error in an individual flight plan, bringing the whole system down with it (along with the backup system since it was running the same code).

> he system can't recover from an error in an individual flight plan, bringing the whole system down with it From the system's POV maybe this is the right way to resolve the problem. Could masking the failure by obscuring this flight's waypoint problem have resulted in a potentially conflicting flight not being tracked among other flights? If so, maybe it's truly urgent enough to bring down the system and force the h…

It seems fundamentally unreasonable for the flight processing system to entirely shut itself down just because it detected that one flight plan had corrupt data. Some degree of robustness should be expected from this system IMO.

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#82

Post by throw0101a »

CrimsonCape wrote: post #65
steeeeeve wrote: post #15

You know there's a software engineer somewhere that saw this as a potential problem, brought up a solution, and had that solution rejected because handling it would add 40 hours of work to a project.

C dev: "You are telling me that the three digit codes are not globally unique??? And now we have to add more bits to the struct?? That's going to kill our perfectly optimized bit layout in memory! F***! This whole app is going to sh**"

> C dev: "You are telling me that the three digit codes are not globally unique???

They are understood not to be. They are generally known to be regionally unique.

The "DVL" code is unique with-in FAA/Transport Canada control, and the "DVL" is unique with-in EASA space.

There are pre-defined three-letter codes:

* https://en.wikipedia.org/wiki/IATA_airport_code

And pre-defined four-letter codes:

* https://en.wikipedia.org/wiki/ICAO_airport_code

There are also five-letter names for major route points:

* https://data.icao.int/icads/Product/View/98

* https://ruk.ca/content/icao-icard-and-5lnc-how-those-5-lette...

If there are duplicates there is a resolution process:

* https://www.icao.int/WACAF/Documents/Meetings/2014/ICARD/ICA...

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#83

Post by pests »

snakeyjake wrote: post #46
jp57 wrote: post #31

Earlier quoted context omitted.

Sorry no. One interpretation, especially on this site, is that the problem was some kind of database bug, maybe where the same location was entered twice and a tiny location error ended up creating two locations. I expect that out of any random sample 500 million literate and mentally healthy English speakers, more than 450 million of them are totally unaccustomed to thinking about nautical miles ever. Even people in…

[flagged]

I assumed nanometers.

I see it every day in chip/design/a few others. I was confused how I caused an issue with the points being so close.

I only realized NM means nautical miles only due to these comments.

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#84

Post by amiga386 »

lostlogin wrote: post #75
crote wrote: post #66

Earlier quoted context omitted.

Coming up with a globally unique waypoint system is trivial. Convincing the aviation industry to spend many hundreds of millions of dollars to change a core data type used in just about every single aviation-related system, in order to avoid triggering rare once-a-decade bugs? That's a lot harder.

> That's a lot harder. I wonder what 1,500 cancelled flights and 700,000 disrupted passengers adds up to in cost? And that’s just this one incident.

...an incident where they didn't parse the data as other systems already parsed the data.

It sounds like the solution is better validation and test suites for the existing scheme, not a new less-ambiguous scheme

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#85

Post by em-bee »

ryandrake wrote: post #28
steeeeeve wrote: post #15

You know there's a software engineer somewhere that saw this as a potential problem, brought up a solution, and had that solution rejected because handling it would add 40 hours of work to a project.

... or there's a software engineer somewhere who simply assumed that three letter navaid identifiers were globally unique, and baked that assumption into the code. I guess we now need a "Falsehoods Programmers Believe About Aviation Data" site :)

or falsehoods programmers believe about global identifiers

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#86

Post by amiga386 »

This is old news, but what's new news is that last week, the UK Civil Aviation Authority openly published its Independent Review of NATS (En Route) Plc's Flight Planning System Failure on 28 August 2023 https://www.caa.co.uk/publication/download/23337 (PDF)

Let's look at point 2.28: "Several factors made the identification and rectification of the failure more protracted than it might otherwise have been. These include:

• The Level 2 engineer was rostered on-call and therefore was not available on site at the time of the failure. Having exhausted remote intervention options, it took 1.5 hours for the individual to arrive on-site to perform the necessary full system re-start which was not possible remotely.

• The engineer team followed escalation protocols which resulted in the assistance of the Level 3 engineer not being sought for more than 3 hours after the initial event.

• The Level 3 engineer was unfamiliar with the specific fault message recorded in the FPRSA-R fault log and required the assistance of Frequentis Comsoft to interpret it.

• The assistance of Frequentis Comsoft, which had a unique level of knowledge of the AMS-UK and FPRSA-R interface, was not sought for more than 4 hours after the initial event.

• The joint decision-making model used by NERL for incident management meant there was no single post-holder with accountability for overall management of the incident, such as a senior Incident Manager.

• The status of the data within the AMS-UK during the period of the incident was not clearly understood.

• There was a lack of clear documentation identifying system connectivity.

• The password login details of the Level 2 engineer could not be readily verified due to the architecture of the system."

WHAT DOES "PASSWORD LOGIN DETAILS ... COULD NOT BE READILY VERIFIED" MEAN?

EDIT: Per NATS Major Incident Investigation Final Report - Flight Plan Reception Suite Automated (FPRSA-R) Sub-system Incident 28th August 2023 https://www.caa.co.uk/publication/download/23340 (PDF) ... "There was a 26-minute delay between the AMS-UK system being ready for use and FPRSA-R being enabled. This was in part caused by a password login issue for the Level 2 Engineer. At this point, the system was brought back up on one server, which did not contain the password database. When the engineer entered the correct password, it could not be verified by the server. "

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#87

Post by SilasX »

s_tec wrote: post #54
snakeyjake wrote: post #46

Earlier quoted context omitted.

[flagged]

I assumed the post title meant nanometers. Why? Floating-point rounding bugs. A nanometer is about 9e-15 degrees of latitude, which is right about where a double-precision floating point number runs out of digits. So, if a piece of software uses exact `==` equality, it could easily have a bug where two positions 3600 nanometers apart are seen as being different, even though they should be treated as the same.

Thank you. People can be very bad about judging which scenarios are truly implausible.

Here’s a previous thread where someone thought it was absurd that there could exist native English speakers who don’t regularly go shopping, and treated that supposed impossibility as a huge “checkmate”!

https://news.ycombinator.com/item?id=32625340

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#88

Post by [deleted] »

snakeyjake wrote: post #46
jp57 wrote: post #31

Earlier quoted context omitted.

Sorry no. One interpretation, especially on this site, is that the problem was some kind of database bug, maybe where the same location was entered twice and a tiny location error ended up creating two locations. I expect that out of any random sample 500 million literate and mentally healthy English speakers, more than 450 million of them are totally unaccustomed to thinking about nautical miles ever. Even people in…

[flagged]

[deleted]

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#89

Post by Arainach »

telgareith wrote: post #69
jrochkind1 wrote: post #33

I don't know how long that failure mode has been in place or if this is relevant, but it makes me think of analogous times I've encountered similar: When automated systems are first put in place, for something high risk, "just shut down if you see something that may be an error" is a totally reasonable plan. After all, literally yesterday they were all functioning without the automated system, if it doesn't seem to b…

Dig into the OpenZFS 2.2.0 data loss bug story. There was at least one ticket (in FreeBSD) where it cropped up almost a year prior and got labeled "look into layer," but it got closed. I'm aware closing tickets of "future investigation" tasks when it seems to not be an issue any longer is common. But, it shouldnt be.

>it shouldnt be

Software can (maybe) be perfect, or it can be relevant to a large user base. It cannot be both.

With an enormous budget and a strictly controlled scope (spacecraft) it may be possible to achieve defect-free software.

In most cases it is not. There are always finite resources, and almost always more ideas than it takes time to implement.

If you are trying to make money, is it worth chasing down issues that affect a miniscule fraction of users that take eng time which could be spent on architectural improvements, features, or bugs affecting more people?

If you are an open source or passion project, is it worth your contributors' limited hours, and will trying to insist people chase down everything drive your contributors away?

The reality in any sufficiently large project is that the bug database will only grow over time. If you leave open every old request and report at P3, users will grow just as disillusioned as if you were honest and closed them as "won't fix". Having thousands of open issues that will never be worked on pollutes the database and makes it harder to keep track of the issues which DO matter.

Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code

#90

Post by nightowl_games »

steeeeeve wrote: post #15

You know there's a software engineer somewhere that saw this as a potential problem, brought up a solution, and had that solution rejected because handling it would add 40 hours of work to a project.

I don't know that and I don't like this assumption that only 'managers' make mistakes, or that software engineers are always right. I thinks needlessly adversarial, biased and largely incorrect.
Post reply on HN