Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

371–380 of 459 posts

Re: UK air traffic control meltdown

#371

Post by tim333 »

nwallin wrote: post #237
sam0x17 wrote: post #163

Earlier quoted context omitted.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

1. Pilots occasionally have to fat finger them into ruggedized I/O devices and read them off to ATC over radios. 2. These are defined by the various regional aviation authorities. The US FAA will define one list, (and they'll be unique in the US) the EU will have one, (EASA?) etc. The AA965 crash (1995-12-20) was due to an aliased waypoint name. Colombia had two waypoints with the same name within 150 nautical miles…

You could maybe make them globally unique by adding the country where appropriate like we do with Paris, France vs Paris, Texas? And not using the same name twice in the same country.

Re: UK air traffic control meltdown

#372

Post by iraqmtpizza »

awestroke wrote: post #369
chii wrote: post #350

Earlier quoted context omitted.

if this is true, then would it be a better investment to have the 2nd team produce a fuzz testing/systematic testing mechanism instead of producing a secondary copy of the same system? In fact, make it adversarial testing such that this team is rewarded (may be financially) if mistakes or problems are found from the 1st team's program.

Such incentives can lead to reduced collaboration. If I get paid every time you make mistakes, I won't want you to get better at your job

the whole point is that they're not collaborating so as to avoid cross-contamination. also you don't get paid unless and until you identify the mistake. if you decrease the reward over time, there is an additional incentive to not sit on the information

Re: UK air traffic control meltdown

#374

Post by iudqnolq »

dhx wrote: post #335
WalterBright wrote: post #322

> the backup system applied the same logic to the flight plan with the same result Oops. In software, the backup system should use different logic. When I worked at Boeing on the 757 stab trim system, there were two avionics computers attached to the wires to activate the trim. The attachment was through a comparator, that would shut off the authority of both boxes if they didn't agree. The boxes were designed with:…

This would have been a 2oo2 system where the pilot becomes the backup. 2oo2 systems are not highly available. Air traffic control systems should at least be 2oo3[1] (3 systems independently developed of which 2 must concur at any given time) so that a failure of one system would still allow the other two to continue operation without impacting availability of the aviation industry. Human backup is not possible becaus…

In this case the problem was choosing an excessively naive algorithm. I'm very inexperienced but that seems to me like the solution would be to spend a bit more money on reviewing the one implementation rather than writing two new ones from scratch.

Re: UK air traffic control meltdown

#375

Post by iudqnolq »

pmarreck wrote: post #363
wlonkly wrote: post #330

Earlier quoted context omitted.

No, and by my understanding it can't be, as the algorithm is now permanent. But it's worse than that, there are confusables within small distances of each other: https://cybergibbons.com/security-2/why-what3words-is-not-su... https://w3w.me.ss/

Since the app gives you the words to say, and translates those back to coordinates on the receiving end, in theory they could alter the word list, at the cost of making any written-down version obsolete. Maybe they should release a new service called What4ActuallyVettedWordsAndWordCombinations ;)

Aren't they trying to turn their word list into a subscription service? Obsoleting paper copies might be a feature.

Re: UK air traffic control meltdown

#376

Post by iudqnolq »

hn_throwaway_99 wrote: post #168
adrianmonk wrote: post #131

Earlier quoted context omitted.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

I agree with the general sentiment "if you see an unexpected error, STOP", but I don't really think that applies here. That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processi…

I'm pretty inexperienced, but I'm starting to learn the hard way that it takes more discipline to add more complex error recovery. (Just recently my implementation of what you're suggesting - limiting the blast radius of server side errors - meant all my tests were passing with a logged error I missed when I made a typo)

Considering their level 1 and 2 support techs couldn't access the so-called "low level" logs with the actual error message it's not clear to me they'd be able to keep up with a system with more complicated failure states. For example, they'd need to make sure that every plan rejected by the computer is routed to and handled by a human.

Re: UK air traffic control meltdown

#377

Post by justinclift »

mannykannot wrote: post #356
dhx wrote: post #335

Earlier quoted context omitted.

This would have been a 2oo2 system where the pilot becomes the backup. 2oo2 systems are not highly available. Air traffic control systems should at least be 2oo3[1] (3 systems independently developed of which 2 must concur at any given time) so that a failure of one system would still allow the other two to continue operation without impacting availability of the aviation industry. Human backup is not possible becaus…

> Air traffic control systems should at least be 2oo3... Human backup is not possible because of human resourcing and complexity. But this was a 1oo1 system, and the human backup handled it well enough: a lot of people were inconvenienced, but there were no catastrophes, and (AFAIK) nothing that got close to being one. As for the benefits of independent development: it might have helped, but the chances of this being…

> But this was a 1oo1 system, and the human backup handled it well enough ...

Heh, a hundred million pound outage. ;)

True, no-one seems to have died from it directly though.

Re: UK air traffic control meltdown

#378

Post by d1sxeyes »

fbdab103 wrote: post #204
d1sxeyes wrote: post #156

Earlier quoted context omitted.

> It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day. This is very possible. I know of a guy who does (or at least a few years ago did) 24x7 365 on-call for a piece of mission (although not safety) critical aviation software. Most of his calls were fixing AWBs quickly because otherwise planes would need to take off empty or los…

Please tell me this guy is now wealthy beyond imagination and living a life of leisure?

I would love to. But it wouldn’t be true.

Re: UK air traffic control meltdown

#379

Post by WalterBright »

borissk wrote: post #336
WalterBright wrote: post #322

> the backup system applied the same logic to the flight plan with the same result Oops. In software, the backup system should use different logic. When I worked at Boeing on the 757 stab trim system, there were two avionics computers attached to the wires to activate the trim. The attachment was through a comparator, that would shut off the authority of both boxes if they didn't agree. The boxes were designed with:…

Different teams often make the same mistake. The system you describe is not perfect, but makes sense.

I neglected to mention there was a third party that reviewed the algorithms to verify they weren't the same.

Nothing is perfect, though, and the pilot is the backup for failure of that system. I.e. turn off the stab trim system.

Re: UK air traffic control meltdown

#380

Post by circular_logic »

What I still don't understand is how flight plans get approved?

In my mind they would only be approved once all involved countries review and process the plan. That way we don't need this ridiculous idea of failing safe on the whole uk airspace for a single error.

That day a single flight plan could have been rejected, perhaps just resubmitted and the bug quietly fixed in the background

Post reply on HN