Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

241–250 of 459 posts

Re: UK air traffic control meltdown

#241

Post by sheepshear »

lbriner wrote: post #76

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same reason. It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do. The bit…

Failing over is correct because there's no way to discern that the hardware is not at fault. They should have designed a better response to the second failure to avoid the knock-on effects.

Re: UK air traffic control meltdown

#242

Post by burntwater »

drachir91 wrote: post #207

What ticked me is that when the primary system threw in the towel, an EXACT SAME system took over and ran the exact same code on the exact same data as the primary. I know that with code and algorithms it's not always the case but even then you know what doing the same thing over and over expecting different results defines... Yes, it can be argued that the software should've had more graceful failure modes and this…

I'm wondering if the backup system could have a delayed queue; say, 30 seconds behind. If the primary fails, and exactly 30 seconds later the secondary system fails, you have reasonable assurance that it was queue input that caused the failure. Rollback to the last successful queue input, skip and flag the suspect input, and see if the next input is successful.

Re: UK air traffic control meltdown

#243

Post by jeffrallen »

zaphar wrote: post #149
d1sxeyes wrote: post #147

Earlier quoted context omitted.

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

I'm no aviation safety controls expert but it seems to me that there are two types of controls that should be in place: 1. Process controls: What do we do when this breaks for any reason. 2. Engineering controls: What can we do to keep this from breaking in the first place? Both of them seem to be somewhat essential for a truly safe system.

One or more of three results can come from the engineering exercise of trying to keep something from breaking in the first place:

1. You could know the solution, but it would be too heavy.

2. You could know the solution, but it would include more parts, each of which would need the same process on it, and the process might fail the same way

3. You miss something and it fails anyway, so your "what if this fails" path better be well rehearsed and executed.

Real engineering is facing the tradeoffs head on, not hand waving them away.

Re: UK air traffic control meltdown

#244

Post by lagt_t »

Rochus wrote: post #67

This is apparently just an opinion, no additional inside information than we had from the report ( https://news.ycombinator.com/item?id=37401981 ), isn't it? EDIT: downvoting this question instead of responding is a pretty strange reaction.

Dude this isn't reddit dont worry about the votes.

Re: UK air traffic control meltdown

#245

Post by crabbone »

c7DJTLrn wrote: post #60

This is an interesting engineering problem and I'm not sure what the best approach is. Fail safe and stop the world, or keep running and risk danger? I imagine critical systems like trading/aerospace have this worked out to some degree.

There isn't and cannot be a preference to either one. It always depends on what the system is doing and what the consequences would be... Pacemaker cannot "fail safe" for example, under no circumstances. It's meaningless to consider such cases. But if escalation to a human operator is possible, then it will also depend on how the system is meant to be used. In some cases it's absolutely necessary that the system doesn't try to handle errors (eg. if say a patient is in a CT machine -- you always want to stop to, at least, prevent more radiation), but in the situation like the one with the flight control -- my guess is that you want the system to keep trying while alerting the human operator.

But then it can also depend on what's in the contract and who will get the blame for the system functioning incorrectly. My guess here is that failing w/o attempting to recover was, while an overkill, a safer strategy than to let eg. two airplanes be scheduled for the same path (and potentially collide).

Re: UK air traffic control meltdown

#246

Post by andrewaylett »

airstrike wrote: post #173
raverbashing wrote: post #154

Earlier quoted context omitted.

And that's why I never (or very rarely) put "this should never happen" exceptions anymore in my code Because you eventually figure out that, yes, it does happen

This here is the true takeaway. The bar for writing "this should never happen" code must be set so impossibly high that it might as well be translated into "'this should never happen' should never happen"

The problem with that is that most programming languages aren't sufficiently expressive to be able to recognise that, say, only a subset of switch cases are actually valid, the others having been already ruled out. It's sometimes possible to re-architect to avoid many of this kind of issue, but not always.

What you're often led to is "if this happens, there's a bug in the code elsewhere" code. It's really hard to know what to do in that situation, other than terminate whatever unit of work you were trying to complete: the only thing you know for sure is that the software doesn't accurately model reality.

In this story, there obviously was a bug in the code. And the broken algorithm shouldn't have passed review. But even so, the safety critical aspect of the complete system wasn't compromised, and that part worked as specified -- I suspect the system behaviour under error conditions was mandated, and I dread to think what might have happened if the developers (the company, not individuals) were allowed to actually assume errors wouldn't happen and let the system continue unchecked.

Re: UK air traffic control meltdown

#247

Post by Rochus »

lagt_t wrote: post #244
Rochus wrote: post #67

This is apparently just an opinion, no additional inside information than we had from the report ( https://news.ycombinator.com/item?id=37401981 ), isn't it? EDIT: downvoting this question instead of responding is a pretty strange reaction.

Dude this isn't reddit dont worry about the votes.

Since it's not Reddit but HN, it's all the stranger to dismiss a perfectly legitimate question. But times and mores seem to change much faster than I realize.

Re: UK air traffic control meltdown

#248

Post by hermitcrab »

dopidopHN wrote: post #227
hermitcrab wrote: post #197

>"in typical Mail Online reporting style: "Did blunder by French airline spark air traffic control issues?" The Daily Mail is a horrible, right-wing paper in the UK that blames 'foreigners' for everything. Particularly the French. Out of curiosity, is there a corresponding French paper that blames the English or the British for everything?

French here, as much as I wish It was the case for comical effect… I don’t think so. Our right wing press is also desperately economically liberal so anything privately run is inherently better. Maybe radio stations? Honestly, major respect to the daily mail for those snarky attacks that keep up the good spirits between our two countries. It’s maybe the food or the weather that make them aggro ? Idk, but don’t worry,…

>major respect to the daily mail for those snarky attacks

There is really nothing to like or respect about the Daily Mail. https://www.globaljustice.org.uk/blog/2017/10/horrible-histo...

>our former colony across the channel

Touché! ;0)

Re: UK air traffic control meltdown

#249

Post by cratermoon »

slt2021 wrote: post #66
cratermoon wrote: post #5

Earlier quoted context omitted.

> why could the system not put the failed flight plan in a queue Because it doesn't look at the data as a "flight plan" consisting of "way points" with "segments" along a "route" that has any internal self-consistency. It's a bag of strings and numbers that's parsed and the result passed along, if parsing is successful. If not, give up. In this case fail the entire system and take it out of production. Airline indust…

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

I never said it was a good ETL system. Heck, I don't even know if the specs for it even specifies what to do with a bad record - there are at least 300 pages detailing the system. Looking around at other stories, I see repeated mentions of how the circumstances leading to this failure are supposedly extremely rare, "one in 15 million" according to one official[1]. But at 100,000 flights/day (estimated), this kind situation would occur, statistically, twice a year.

1 https://news.sky.com/story/major-flights-disruption-caused-b...

Re: UK air traffic control meltdown

#250

Post by blitzar »

f1shy wrote: post #174
sam0x17 wrote: post #163

Earlier quoted context omitted.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

The names have to be entered manually by pilots, if e.g. they change the route. They have to be transmitted over the air by humans. So they must be short ans simple.

Clippy: It looks like you are trying to enter a non unique navigation point, did you mean the one in France or the one in Australia?
Post reply on HN