Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

321–330 of 459 posts

Re: UK air traffic control meltdown

#321

Post by theptip »

crabbone wrote: post #232
theptip wrote: post #169

Earlier quoted context omitted.

None of this really argues against fuzz testing; even with completely bogus/malformed flight plans, it shouldn't be possible for a dead letter to take down the entire system. And, since it's translating between an upstream and downstream format (and all the validation is done when ingesting the upstream), you probably want to be sure anything that is valid upstream is also valid downstream. It's true that fuzz testin…

This is not against the principle of fuzz testing. This is to say that the author doesn't really know the reality of testing and is very quick to point fingers. It's easy to tell in retrospect that this particular aspect should've been tested. It's basically impossible to find such defects proactively.

Easy for me to say in retrospect, but IMO this is a textbook example of where you should reach for fuzz testing; it’s basically protocol parsing, you have a well-known text format upstream and you need to ensure your system can parse all well-formed protocol messages and at very least not crash if a given message is invalid in your own system.

Similarly with a message queue, handling dead letters is textbook stuff, and you must have system tests to verify that poison pills do not break your queue.

I did not think the author was setting unreasonable expectations for the a priori testing regime. These are common best practices.

Re: UK air traffic control meltdown

#322

Post by WalterBright »

> the backup system applied the same logic to the flight plan with the same result

Oops. In software, the backup system should use different logic. When I worked at Boeing on the 757 stab trim system, there were two avionics computers attached to the wires to activate the trim. The attachment was through a comparator, that would shut off the authority of both boxes if they didn't agree.

The boxes were designed with:

1. different algorithms

2. different programming languages

3. different CPUs

4. code written by different teams with a firewall between them

The idea was that bugs from one box would not cause the other to fail in the same way.

Re: UK air traffic control meltdown

#323

Post by [deleted] »

DyslexicAtheist wrote: post #278
nwallin wrote: post #237

Earlier quoted context omitted.

1. Pilots occasionally have to fat finger them into ruggedized I/O devices and read them off to ATC over radios. 2. These are defined by the various regional aviation authorities. The US FAA will define one list, (and they'll be unique in the US) the EU will have one, (EASA?) etc. The AA965 crash (1995-12-20) was due to an aliased waypoint name. Colombia had two waypoints with the same name within 150 nautical miles…

> Pilots occasionally have to fat finger them into ruggedized I/O devices you're saying what-3-words (W3W) is unsuitable for safety critical applications ? /s

[deleted]

Re: UK air traffic control meltdown

#324

Post by WalterBright »

dharmab wrote: post #233
epanchin wrote: post #215

Earlier quoted context omitted.

What three words would be a better solution than a guid, as transmittable over radio.

W3W contains homonyms and words that are easily confused by non-native english speakers. Often within just a few KM. The latter is why ATC uses "niner", to avoid confusing "nine" and "nein". Talk to someone deep in the GIS rabbit hole and you'll get a rant about how bad W3W is: https://cybergibbons.com/security-2/why-what3words-is-not-su...

Wernher von Braun polled his reports for whether the rocket was reliable or not. Each engineer replied "nein".

Von Braun reported that the rocket had six nines probability of success.

Re: UK air traffic control meltdown

#325

Post by [deleted] »

d1sxeyes wrote: post #147
lbriner wrote: post #76

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same reason. It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do. The bit…

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

[deleted]

Re: UK air traffic control meltdown

#327

Post by wavemode »

WalterBright wrote: post #322

> the backup system applied the same logic to the flight plan with the same result Oops. In software, the backup system should use different logic. When I worked at Boeing on the 757 stab trim system, there were two avionics computers attached to the wires to activate the trim. The attachment was through a comparator, that would shut off the authority of both boxes if they didn't agree. The boxes were designed with:…

First thought that came to my mind as well when I read it. This failover system seems to be more designed to mitigate hardware failures than software bugs.

Re: UK air traffic control meltdown

#328

Post by WalterBright »

hn_throwaway_99 wrote: post #168
adrianmonk wrote: post #131

Earlier quoted context omitted.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

I agree with the general sentiment "if you see an unexpected error, STOP", but I don't really think that applies here. That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processi…

An "unexpected error" is always a logic bug. The cause of the logic error is not known, because it is unexpected. Therefore, the software cannot determine if it is an isolated problem or a systemic problem. For a systemic problem, shutting down the system and engaging the backup is the correct solution.

Re: UK air traffic control meltdown

#329

Post by WalterBright »

wavemode wrote: post #327
WalterBright wrote: post #322

> the backup system applied the same logic to the flight plan with the same result Oops. In software, the backup system should use different logic. When I worked at Boeing on the 757 stab trim system, there were two avionics computers attached to the wires to activate the trim. The attachment was through a comparator, that would shut off the authority of both boxes if they didn't agree. The boxes were designed with:…

First thought that came to my mind as well when I read it. This failover system seems to be more designed to mitigate hardware failures than software bugs.

I also understand that it is impractical to implement the ATC system software twice using different algorithms. The software at least checked for an illogical state and exited, which was the right thing to do.

A fix I would consider is to have the inputs more thoroughly checked for correctness before passing them on to the ATC system.

Re: UK air traffic control meltdown

#330

Post by wlonkly »

jonny_eh wrote: post #317
joshuamorton wrote: post #294

Earlier quoted context omitted.

I'm trying to imagine someone ensuring differentiation between minimums.unsettled.depends (Idaho), minimums.unsettled.depend (Alaska), minimums.unsettles.depend (Spain), and minimum.unsettles.depend (Russia) while typing them in on a t-9 style keypad with a 7 figure display in turbulence.

That seems like a huge flaw in their system, has it never been addressed?

No, and by my understanding it can't be, as the algorithm is now permanent.

But it's worse than that, there are confusables within small distances of each other:

https://cybergibbons.com/security-2/why-what3words-is-not-su...

https://w3w.me.ss/

Post reply on HN