Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

391–400 of 459 posts

Re: UK air traffic control meltdown

#391

Post by bbx »

algas wrote: post #281
Gud wrote: post #99

A day I don't want to remember. Took me 15 hours to reach my destination instead of 2. Had to take train, bus, then train again. 30 minutes after I had booked my tickets, everything was fully booked for two days.

I waited in the airport for 6 hours before learning that my flight was cancelled, and had to rebook... I was flying to New York to see my family, so I didn't really have any alternate transportation options!

That's a shame, sorry to hear that. I got more lucky: I had to wait for 6 hours too but my flight suddenly resumed (must have been one of the first few). I didn't have any alternatives to go home either so I feel for all of those stuck in a foreign country.

Re: UK air traffic control meltdown

#392

Post by anentropic »

sheepshear wrote: post #241
lbriner wrote: post #76

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same reason. It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do. The bit…

Failing over is correct because there's no way to discern that the hardware is not at fault. They should have designed a better response to the second failure to avoid the knock-on effects.

I don't think anything in this incident pointed to a hardware fault

The software raised an exception because a "// TODO: this should never happen" case happened

A hardware fault would look like machines not talking to each other or corrupted data file unreadable

Re: UK air traffic control meltdown

#393

Post by Neil44 »

fennecfoxy wrote: post #390

"UK air traffic control: inquiry into whether French error caused failure" Of course bloody not. How is it a French airline's fault when it's a UK system? Systems like this should be foolproof with redundancies. If one entry is bad reject it and carry on, even.

Well if you want to get all nationalistic, the software was Austrian.

Re: UK air traffic control meltdown

#394

Post by anentropic »

worik wrote: post #198

I heard in the news that this was caused by a "bad flight plan". It is clear, even without any more information than that, it was a software failure (bad flight plan?) It will be interesting to see if Frequentis has to pay a price for causing this

Yeah it seems clear from the report that NATS published it wasn't a bad flight plan at all... the plan was valid to the relevant specifications

But the specifications allow ambiguities (non-unique waypoint ids) and the software did not handle this particular ambiguity correctly

Re: UK air traffic control meltdown

#395

Post by anentropic »

cja wrote: post #199

Every system I've ever made has better error reporting that that one. Even those that only I use. First thing I get working in a new project is the system to tell me when something fails and to help me understand and fix the problem quickly. I then use that system throughout development such that it works very well in production. I'd love to talk to the people who made the system discussed in the article. Is one of t…

Yes it seems incredibly lame error reporting that they had to spend hours contacting the original vendor (to "analyse low-level software logs") just to find out which flight plan had crashed the system

Re: UK air traffic control meltdown

#396

Post by anentropic »

crabbone wrote: post #167

I want to comment specifically on: > The software and system are not properly tested. Followed by suggesting to do fuzzing tests. * Automatically generating valid flight paths is somewhat hard (and you'd have to know which ones are valid because the system, apparently, is designed to also reject some paths). It's also possible that such a generator would generate valid but improbable flight paths. There's probably an…

This all sounds like exactly the stuff that fuzzing or property-based testing is good for

And if the functionality is "buried deep inside other code with no direct or easy way to extricate it from its surrounding" making it hard to test then that's just a further symptom of badly designed software in this case

Re: UK air traffic control meltdown

#397

Post by virtue3 »

iudqnolq wrote: post #374
dhx wrote: post #335

Earlier quoted context omitted.

This would have been a 2oo2 system where the pilot becomes the backup. 2oo2 systems are not highly available. Air traffic control systems should at least be 2oo3[1] (3 systems independently developed of which 2 must concur at any given time) so that a failure of one system would still allow the other two to continue operation without impacting availability of the aviation industry. Human backup is not possible becaus…

In this case the problem was choosing an excessively naive algorithm. I'm very inexperienced but that seems to me like the solution would be to spend a bit more money on reviewing the one implementation rather than writing two new ones from scratch.

you would be very surprised how difficult avionics are from even a fundamental level.

I'll provide a relatively simple example.

Just even attempting to design a starfox game clone where the ship goes towards the mouse cursor using euler angles will almost immediately result in gimbol lock and your starfighter locking up tighter than unlubricated car engine going 100mph and unable to move. [0]

The standard solution in games(or at least what I used) has been to use quaternions [1] (Hamilton defined a quaternion as the quotient of two directed lines in a three-dimensional space,[3] or, equivalently, as the quotient of two vectors.) So you essentially dump your 3D coordinate into the 4D quaternion coordinate, apply your matrix rotations, then convert back to 3D space and apply your rotations/transforms.

This was literally just to get my little space ship to go where my mouse cursor was on the screen without it locking up.

So... yeah, I cannot even begin to imagine the complexity of what a Boeing 757 (let alone a 787) is doing under the hood to deal with reality and not causing it to brick up and fall out of the sky.

[0] https://math.stackexchange.com/questions/8980/euler-angles-a... [1] https://en.wikipedia.org/wiki/Quaternion

Re: UK air traffic control meltdown

#398

Post by iudqnolq »

virtue3 wrote: post #397
iudqnolq wrote: post #374

Earlier quoted context omitted.

In this case the problem was choosing an excessively naive algorithm. I'm very inexperienced but that seems to me like the solution would be to spend a bit more money on reviewing the one implementation rather than writing two new ones from scratch.

you would be very surprised how difficult avionics are from even a fundamental level. I'll provide a relatively simple example. Just even attempting to design a starfox game clone where the ship goes towards the mouse cursor using euler angles will almost immediately result in gimbol lock and your starfighter locking up tighter than unlubricated car engine going 100mph and unable to move. [0] The standard solution in…

I don't think we're talking about that kind of software, though. This big was in code that needs to parse a line defined by named points and then clip the line to the portion in the UK. Not trivial, but I can imagine writing that myself.

But regardless the more complex the code the worse idea it is to maintain three parallel implementations, if you won't/can't afford to do it properly

Re: UK air traffic control meltdown

#399

Post by d1sxeyes »

zaphar wrote: post #149
d1sxeyes wrote: post #147

Earlier quoted context omitted.

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

I'm no aviation safety controls expert but it seems to me that there are two types of controls that should be in place: 1. Process controls: What do we do when this breaks for any reason. 2. Engineering controls: What can we do to keep this from breaking in the first place? Both of them seem to be somewhat essential for a truly safe system.

The engineering controls don't independently make systems safe, they make things more reliable and cost-effective, and hopefully reduce the number of times the process controls kick in.

The process controls do however independently make things safe.

The reason for this is that there are 'unknown unknowns'—we accept that our knowledge and skills are imperfect, and there may be failures that occur which could have been eliminated with the proper engineering controls, but we, as imperfect beings and organisations, did not implement the engineering controls because we did not identify this possible failure mode.

There are also known errors, where the cost of implementing engineering controls may simply outweigh the benefits when adequate process controls are in place.

Re: UK air traffic control meltdown

#400

Post by jwr »

wcarss wrote: post #341
dhx wrote: post #335

Earlier quoted context omitted.

This would have been a 2oo2 system where the pilot becomes the backup. 2oo2 systems are not highly available. Air traffic control systems should at least be 2oo3[1] (3 systems independently developed of which 2 must concur at any given time) so that a failure of one system would still allow the other two to continue operation without impacting availability of the aviation industry. Human backup is not possible becaus…

This reminds me of a backwoods hike I took with a friend some years back. We each brought a compass, "for redundancy", but it wasn't until we were well underway that we noticed our respective compasses frequently disagreed. We often wished we had a third to break the tie!

My grandfather was working with Stanisław Skarżyński, who was preparing for his first crossing of the Atlantic in a lightweight airplane (RWD-5bis, 450kg empty weight) in 1933.

They initially mounted two compasses in the cockpit, but Skarżyński taped one of them over so that it wasn't visible, saying wisely that if one fails, he will have no idea which one is correct.

Post reply on HN