Full report link: https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
Thanks. One thing that I don't quite understand is how do new waypoints get added as part of the conversion of a flight plan from ICAO4444 to ADEXP format? Does it do some kind of interpolation? Also, it appears the error was caused by the algorithm selecting two waypoints with the same identifier as the entry and exit points into UK airspace for this particular flight plan. But it also says non-unique waypoints shou…
I'm just speculating as to what points are actually added but typically the flight plan includes 'airways' in addition to waypoints. Here's an example of a plan for London Gatwick (EGKK) to Edinburgh (EGPH):
In this case, BPK and INPIP are two waypoints (one in the south of England and one in the north). UN601 is an airway that connects these two waypoints. The airway represents a ton of other waypoints between BPK and INPIP but that you don't need to specify manually in the flight plan. I suspect the additional waypoints added are the splitting of the airway into the underlying waypoints - but as I say it's speculation :-)
Full report link: https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
" An FPRSA sub-system has existed in NATS for many years and in 2018 the previous FPRSA subsystem was replaced with new hardware and software manufactured by Frequentis AG, one of the leading global ATC System providers. " " At this point with both the primary and backup FPRSA-R sub-systems having failed safely the FPRSA-R was no longer able to automatically process flight plans. It required restoration to normal ser…
> How can a primary AND it's backup system fail safely??? Who specified this?
All safety critical systems are specified to halt instead of performing undefined behavior, if they encounter something that cannot be processed. An unsafe failure would be entering undefined behaviour.
What would you have specified differently, that would be safer?
A backup is primarily there in case of hardware failures or for maintenance. If it behaves differently to the primary then something is wrong.
Can you explain how and why you would expect a backup system running identical software to behave differently?
> This is the root cause of the incident. We can therefore rule out any cyber related contribution to this incident. They were hacked. Obviously this is the first time this flight path has been filed, otherwise it would have crashed earlier. Phone Phreaking, whilst not technically a cyber attack, is still a form of hacking of phone systems. And email bombs exist which take out email servers and readers, and zip bombs…
" An FPRSA sub-system has existed in NATS for many years and in 2018 the previous FPRSA subsystem was replaced with new hardware and software manufactured by Frequentis AG, one of the leading global ATC System providers. " " At this point with both the primary and backup FPRSA-R sub-systems having failed safely the FPRSA-R was no longer able to automatically process flight plans. It required restoration to normal ser…
> How can a primary AND it's backup system fail safely??? Who specified this? All safety critical systems are specified to halt instead of performing undefined behavior, if they encounter something that cannot be processed. An unsafe failure would be entering undefined behaviour. What would you have specified differently, that would be safer? A backup is primarily there in case of hardware failures or for maintenance…
I worked in safety-critical ATC projects in engineering and management positions (systems, quality and compliance engineering) for a decade. ATC systems are supposed to not fail, even under adverse conditions. Where high availability is required for safety reasons, redundant architectures is one of the options. Apparently the "backup system" was conceived for this purpose. According to the report (page 17) the responsible subsystem suffered from a "critical exception [..] that triggers the conditions that led to the incident", which let both the primary and backup system fail, and has now apparently been fixed. So obviously the system was not supposed to fail on receiving wrong or suspicious flight plan data, and it was apparently pure luck that no such data arrived for five years. To claim that the subsystem (consisting of the primary and backup system) "safely failed" indicates significant gaps in safety management (either faulty safety analyses, faulty specifications, or faulty configuration or software). The report suggests that critical omissions occured at several levels.
> How can a primary AND it's backup system fail safely??? Who specified this? All safety critical systems are specified to halt instead of performing undefined behavior, if they encounter something that cannot be processed. An unsafe failure would be entering undefined behaviour. What would you have specified differently, that would be safer? A backup is primarily there in case of hardware failures or for maintenance…
I worked in safety-critical ATC projects in engineering and management positions (systems, quality and compliance engineering) for a decade. ATC systems are supposed to not fail, even under adverse conditions. Where high availability is required for safety reasons, redundant architectures is one of the options. Apparently the "backup system" was conceived for this purpose. According to the report (page 17) the respon…
For me it's important to consider the 'ATC system' as the whole. The system as a whole did not fail - no planes crashed, flights still flew - but it was in a degraded state with lower than usual throughput. One component of the system did fail (the FPRSA subsystem) and it seems reasonable to me that given layers of the system lean towards unavailability rather than trying to continue to operate in unforeseen circumstances.
The purpose of a backup system is not to prevent failure - it's to improve resiliency of the system as a whole across a set of foreseen and unforeseen faults. Backup systems failing to handle any specific fault is an expected and predicted behavior. Thankfully in this case there was a backup system that prevented a complete shutdown (and, thankfully, any accident) - the manual processing of flight plans.
I worked in safety-critical ATC projects in engineering and management positions (systems, quality and compliance engineering) for a decade. ATC systems are supposed to not fail, even under adverse conditions. Where high availability is required for safety reasons, redundant architectures is one of the options. Apparently the "backup system" was conceived for this purpose. According to the report (page 17) the respon…
For me it's important to consider the 'ATC system' as the whole. The system as a whole did not fail - no planes crashed, flights still flew - but it was in a degraded state with lower than usual throughput. One component of the system did fail (the FPRSA subsystem) and it seems reasonable to me that given layers of the system lean towards unavailability rather than trying to continue to operate in unforeseen circumst…
Missing the availability requirements is a failure.
Safety is not only about human lives, but also about health and property (also e.g. critical financial and other losses, or reputational damage). The present incident has obviously caused considerable damage. We can only hope that the rest of the system does not suffer from similar omissions and that it is not pure coincidence that even worse events occur.
And they shut down the entire system because of one incoherent plan? What a great example of an ingenious high-availability architecture.
> rather than reject the flight plan for an aircraft that may already be in the air I wonder why they can't reject the flight plan for an aircraft that's already in the air? Presumably they have any number of reasons to reject a perfectly valid flight plan that's been submitted yet alone invalid ones there must be a rejection mechanism? The explanation offered by the report (from a quick skim found this on page 9) is…
> I wonder why they can't reject the flight plan for an aircraft that's already in the air?
You need to know everything that may be in the air - if you skip the details of a flight that may be in the air, you risk routing another flight through the same space and the possibility of collision? So if you can't do that safely, the only option is to shut down; existing flights can continue but no new flights can be routed until the anomaly is resolved.
For me it's important to consider the 'ATC system' as the whole. The system as a whole did not fail - no planes crashed, flights still flew - but it was in a degraded state with lower than usual throughput. One component of the system did fail (the FPRSA subsystem) and it seems reasonable to me that given layers of the system lean towards unavailability rather than trying to continue to operate in unforeseen circumst…
Missing the availability requirements is a failure. Safety is not only about human lives, but also about health and property (also e.g. critical financial and other losses, or reputational damage). The present incident has obviously caused considerable damage. We can only hope that the rest of the system does not suffer from similar omissions and that it is not pure coincidence that even worse events occur.
Yeah of course, but success/failure is also not binary. There are degrees of failure, including low-consequence availability issues, high-consequence availability issues, loss of operational safety, 'never events' (e.g. significant loss of life). In this case the system suffered the second of those options. It seems reasonable that design choices may prioritise that type of failure over the later ones in the list.
The first part of this argument is semantics - how do we define failure. The second part is IMHO more important - what decisions are taken with regards to the behavior of subsystems and how they influence overall system degredation. In this case the overall design prevented any loss of operational safety which, to me, is a success.
Missing the availability requirements is a failure. Safety is not only about human lives, but also about health and property (also e.g. critical financial and other losses, or reputational damage). The present incident has obviously caused considerable damage. We can only hope that the rest of the system does not suffer from similar omissions and that it is not pure coincidence that even worse events occur.
Yeah of course, but success/failure is also not binary. There are degrees of failure, including low-consequence availability issues, high-consequence availability issues, loss of operational safety, 'never events' (e.g. significant loss of life). In this case the system suffered the second of those options. It seems reasonable that design choices may prioritise that type of failure over the later ones in the list. Th…
One can talk things up or fix them. As the report (and some comments) suggests, the former is given high priority.
> rather than reject the flight plan for an aircraft that may already be in the air I wonder why they can't reject the flight plan for an aircraft that's already in the air? Presumably they have any number of reasons to reject a perfectly valid flight plan that's been submitted yet alone invalid ones there must be a rejection mechanism? The explanation offered by the report (from a quick skim found this on page 9) is…
> I wonder why they can't reject the flight plan for an aircraft that's already in the air? You need to know everything that may be in the air - if you skip the details of a flight that may be in the air, you risk routing another flight through the same space and the possibility of collision? So if you can't do that safely, the only option is to shut down; existing flights can continue but no new flights can be route…