Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

411–420 of 459 posts

Re: UK air traffic control meltdown

#411

Post by sheepshear »

anentropic wrote: post #392
sheepshear wrote: post #241

Earlier quoted context omitted.

Failing over is correct because there's no way to discern that the hardware is not at fault. They should have designed a better response to the second failure to avoid the knock-on effects.

I don't think anything in this incident pointed to a hardware fault The software raised an exception because a "// TODO: this should never happen" case happened A hardware fault would look like machines not talking to each other or corrupted data file unreadable

Retroactive inspection revealed that it wasn't a hardware failure, but the computer didn't know that at the time, and hardware failure can look like anything, so it was correct to exercise its only option.

Re: UK air traffic control meltdown

#412

Post by fransje26 »

littlestymaar wrote: post #159
rcostin2k2 wrote: post #15

The fact that they blamed the French flight plan already accepted by Eurocontrol proves that they didn't really know how the software works. And here the Austrian company should take part of the blame for the lack of intensive testing.

They blamed the French because they are British, that's it. It's hard to get rid of bad habits.

But, but, but... ...the EU!

Re: UK air traffic control meltdown

#413

Post by javier_e06 »

I worked once with 4G BTS (Base Transceiver Stations) where one of the issues was preventing the errors in the running board to propagate to the backup systems. There was no clean way to do it given the fact the malformed input will eventually reach the backup system producing the same error. The post talks about the system delaying the process to prevent backup up. Perhaps a solution would be going in the other direction having a staging step to prevent compromising the pipeline. Very interesting article.

Re: UK air traffic control meltdown

#414

Post by J8K357R »

Poison Pill! Why on earth would the best failure mode be to cease operating? Just don’t accept the new plan being ingested and tell the person uploading that their plan was rejected. Impact one flight not thousands!

Re: UK air traffic control meltdown

#415

Post by pasc1878 »

J8K357R wrote: post #414

Poison Pill! Why on earth would the best failure mode be to cease operating? Just don’t accept the new plan being ingested and tell the person uploading that their plan was rejected. Impact one flight not thousands!

Ok if the system finds something that it does not understand what should it do - and how does the programmer know it will work?

Re: UK air traffic control meltdown

#416

Post by benrutter »

J8K357R wrote: post #414

Poison Pill! Why on earth would the best failure mode be to cease operating? Just don’t accept the new plan being ingested and tell the person uploading that their plan was rejected. Impact one flight not thousands!

I wondered this- I have absolutely no understanding of what's involved in flight system development, but does anyone know why it doesn't do this?

By contrast, its normal for an API to return 500 if something goes wrong and keep serving other requests. It would seem insane if it crashed out and completely stopped. Any idea why the parallel isn't true for a flight system?

Re: UK air traffic control meltdown

#417

Post by ExoticPearTree »

WalterBright wrote: post #329
wavemode wrote: post #327

Earlier quoted context omitted.

First thought that came to my mind as well when I read it. This failover system seems to be more designed to mitigate hardware failures than software bugs.

I also understand that it is impractical to implement the ATC system software twice using different algorithms. The software at least checked for an illogical state and exited, which was the right thing to do. A fix I would consider is to have the inputs more thoroughly checked for correctness before passing them on to the ATC system.

The system should have just rejected the FPL, notify the admins about the problem and keep working. The admins could have fixed whatever the software could not handle.

The affected flight could have been vectored by ATC if needed to divert from filed FPL.

Way less work and a better doutcome than the “system throws hands in the air and becomes unresponsive”.

Re: UK air traffic control meltdown

#418

Post by dghughes »

dopidopHN wrote: post #227
hermitcrab wrote: post #197

>"in typical Mail Online reporting style: "Did blunder by French airline spark air traffic control issues?" The Daily Mail is a horrible, right-wing paper in the UK that blames 'foreigners' for everything. Particularly the French. Out of curiosity, is there a corresponding French paper that blames the English or the British for everything?

French here, as much as I wish It was the case for comical effect… I don’t think so. Our right wing press is also desperately economically liberal so anything privately run is inherently better. Maybe radio stations? Honestly, major respect to the daily mail for those snarky attacks that keep up the good spirits between our two countries. It’s maybe the food or the weather that make them aggro ? Idk, but don’t worry,…

Isn't that why France has a President the position was created just to blame them? lol

Re: UK air traffic control meltdown

#419

Post by crabbone »

samus wrote: post #409
crabbone wrote: post #256

Earlier quoted context omitted.

Neither formal methods nor fuzzing would've helped if the programmer didn't know that input can repeat. Maybe they just didn't read the paragraph in whatever document describes how this should work and didn't know about it. I didn't have to implement flight control software, but I had to write some stuff described by MIFID. It's a job from hell, if you take it seriously. It's a series of normative documents that expl…

The point of fuzzing is precisely to discover cases that the programmers couldn't think about, and formal methods are useful to discover invariants and assumptions that programmers didn't know they rely on. Furthermore, identifiers from external systems always deserve scepticism. Even UUIDs can be suspect. Magic strings from hell even more so.

Sorry, you missed the point.

If programmer didn't know that repetitions are allowed, they wouldn't appear in the input to the fuzzer as well.

The mistake is too trivial to attribute it to the programmer incompetence / lack of attention. I'd bet my lunch it was because the spec is written in an incomprehensible language, is all over the place in a thousand pages PDF, and the particular aspect of repetition isn't covered in what looks like the main description of how paths are defined.

I've dealt with specs like that. It's most likely the error created by the lack of understanding of the details of the requirements than of anything else. No automatic testing technique would help here. More rigorous and systematic approach to requirement specification would probably help, but we have no tools and no processes to address that.

Re: UK air traffic control meltdown

#420

Post by ExoticPearTree »

sam0x17 wrote: post #163
reactordev wrote: post #28

So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

Because they need to be short, that's why they are 5 letters long. And need to be understood phonetically very quickly by pilots.
Post reply on HN