Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

151–160 of 459 posts

Re: UK air traffic control meltdown

#151

Post by samus »

d1sxeyes wrote: post #147
lbriner wrote: post #76

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same reason. It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do. The bit…

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

It was in a bad state, but in a very inane way: a flight plan in its processing queue was faulty. The system itself was mostly fine. It was just not well-written enough to distinguish an input error from an internal error, and thus didn't just skip the faulty flight plan.

Re: UK air traffic control meltdown

#152

Post by tpmx »

krisoft wrote: post #70
NovemberWhiskey wrote: post #10

This is not the first time this has happened; the phenomenon has even got a name - "poison flight plan".

> the phenomenon has even got a name - "poison flight plan". Maybe, but it must not be a common phrase because your comment is the first result when I search for it. And it is also mentioned in this article: http://www.aero-news.net/subsite.cfm?do=main.textpost&id=ce2... And that's about it? Do you have any other sources?

I think that term was invented four days ago by that article writer. There are four other occurrances before then and they're about PS2 games.

Re: UK air traffic control meltdown

#153

Post by codeulike »

This is a great post. My reading of it:

- waypoint names used around the world are not unique

- as a sortof cludge, "In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced."

- but still you might get the same waypoint name used twice in a route to mean different places

- the software was not written with that possibilty in mind

- route did not compute

- threw 'critical exception' and entered 'maintenance mode' - i.e. crashed

- backup system took over, hit the same bug with the same bit of data, also crashed

- support people have a crap time

- it wasnt until they called the software supplier that they found the low level logs that revealed the cause of the problem

Re: UK air traffic control meltdown

#154

Post by raverbashing »

adrianmonk wrote: post #131
asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

And that's why I never (or very rarely) put "this should never happen" exceptions anymore in my code

Because you eventually figure out that, yes, it does happen

Re: UK air traffic control meltdown

#155

Post by samus »

adrianmonk wrote: post #131
asimpleusecase wrote: post #3

And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

That reasoning is fine, but it rather seems that the programmers triggered this catastrophic "stop the world" error because they were not thorough enough considering all scenarios. As TA expounds, it seems that neither formal methods nor fuzzing were used, which would have gone a long way flushing out such errors.

Re: UK air traffic control meltdown

#156

Post by d1sxeyes »

jandrese wrote: post #119
slt2021 wrote: post #66

Earlier quoted context omitted.

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it. The thing that blows my mind is that this was apparently the…

> It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.

This is very possible. I know of a guy who does (or at least a few years ago did) 24x7 365 on-call for a piece of mission (although not safety) critical aviation software.

Most of his calls were fixing AWBs quickly because otherwise planes would need to take off empty or lose their take-off slot.

Although there had been some “bus factor” planning and mitigation around this guy’s role, it involved engaging vendors etc. and would have likely resulted in a lot of disruption in the short term.

Re: UK air traffic control meltdown

#157

Post by Twirrim »

samus wrote: post #151
d1sxeyes wrote: post #147

Earlier quoted context omitted.

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

It was in a bad state, but in a very inane way: a flight plan in its processing queue was faulty. The system itself was mostly fine. It was just not well-written enough to distinguish an input error from an internal error, and thus didn't just skip the faulty flight plan.

at the risk of nitpicking: "a flight plan in its processing queue was faulty" isn't true, the flight plan was fine. It couldn't process it.

I mention this only because the Daily Mail headline pissed me off with it's usual bullshit foreigner fear mongering crap.

Re: UK air traffic control meltdown

#158

Post by zaphar »

jandrese wrote: post #119
slt2021 wrote: post #66

Earlier quoted context omitted.

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it. The thing that blows my mind is that this was apparently the…

Bad records aren't supposed to be ignored. They are supposed to be looked at by a human who can determine what to do.

Failing the way NATS did means that all future flight plan data including for planes already in the sky are not longer being processed. The safer failure mode was definitely to flag this plan and surface to a human while continuing to process other plans.

Re: UK air traffic control meltdown

#159

Post by littlestymaar »

rcostin2k2 wrote: post #15

The fact that they blamed the French flight plan already accepted by Eurocontrol proves that they didn't really know how the software works. And here the Austrian company should take part of the blame for the lack of intensive testing.

They blamed the French because they are British, that's it. It's hard to get rid of bad habits.
Post reply on HN