Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

281–290 of 459 posts

Re: UK air traffic control meltdown

#281
post #99

A day I don't want to remember. Took me 15 hours to reach my destination instead of 2. Had to take train, bus, then train again. 30 minutes after I had booked my tickets, everything was fully booked for two days.

I waited in the airport for 6 hours before learning that my flight was cancelled, and had to rebook... I was flying to New York to see my family, so I didn't really have any alternate transportation options!

Re: UK air traffic control meltdown

#282

Earlier quoted context omitted.

> an EXACT SAME system took over and ran the exact same code Did you ever work with HA systems? Because this is how they work. It's two copies of the same system intended for the cases when eg. hardware fails, or network partitioning happens etc.

No, I do not. But HA systems work like that because hardware or network failure is what they are designed to guard against, not a latent bug in the software logic. If there's a software bug, both systems will exhibit the same behavior, so HA fails there.

How can you tell whether the problem is due to a software bug or due to a hardware fault though? The software could have thrown the "catastrophic failure, stop the world" exception due to memory corruption.

Re: UK air traffic control meltdown

#283

Earlier quoted context omitted.

My jaw kept dropping with each new bullet point.

Same, is aviation technology really this primitive?

It's worse than you know. Ancient computer systems, non-ASCII character encodings, analog phone lines, and ticker-tape weather.

You'll also be surprised to learn there's still parts of the US where there's no radar or radio coverage with ATC, if flying at lower altitudes. (Heck, there's still a part of the Pacific Ocean that doesn't have ATC service at any altitude.)

Aviation drove a lot of the early developments in networked computing, which also means there's some really old tech in the stack. The globally decentralized nature of it all and it being a life-critical system means it's expensive and complicated to upgrade. (And to be clear, it does get upgraded - but it in a backwards compatible way.) Today's ATC systems need to work with planes built in the 1950s, and talk to ATC units in small countries that still use ancient teletype systems and fax machines.

But yet it's all still incredibly safe, because the technology is there to augment human processes - not replace them. Even if all the technology fails, everything can still be done manually using pen and paper.

Re: UK air traffic control meltdown

#284

Earlier quoted context omitted.

I agree with the general sentiment "if you see an unexpected error, STOP", but I don't really think that applies here. That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processi…

> is essentially totally independent They physically cannot be independent. The system works on an assumption that the flight was accepted and is valid, but it cannot place it. What if it accidentally schedules another flight in the same time and place?

> What if it accidentally schedules another flight in the same time and place?

Flight plans are not responsible for flight separation. It is not their job and nobody uses them for that.

As a first approximation they are used so ATC doesn’t need to ask every airplane every five minute “so flight ABC123 where do you want to go today?”

I’m staring to think that there is a need for a “falsehoods programers believe about aviation” article.

Re: UK air traffic control meltdown

#285

Earlier quoted context omitted.

How exactly would you do that? It’s impossible to map from a dataset of non-unique identifiers to unique identifiers without additional data and heuristics. The mapping is ambiguous by definition. The underlying flight plan standard were all created in an era of low memory machines, and when humans were expected to directly interpret data exactly as the programs represented it internally (because serialisation and de…

If two waypoints have the same name assume its the closest one to the adjacent ones in the route-chain rather than the one 4000 km away

Sounds like a helpful idea with considerable implementation complexity, including the potential for new disastrous failure modes.

Re: UK air traffic control meltdown

#286

Earlier quoted context omitted.

In some sense, yes. Notice that most of the responses to what I've said are immediately negative or dismissive of the idea. If that's the starting point (bad mindset), of course nothing gets fixed and you land where we are today. My initial approach would be to weed out anyone with that point of view before any work took place (the "not HR friendly" part being to be purposefully exclusionary). The only way a problem…

Anyone who has worked with me knows that I have no issue coming in like a wrecking ball in order to make things happen, when necessary. I've also been involved in some of these migration projects. I think your take on the complexity of these projects (and I do mean inherent complexity, not incidental complexity) and the responses you've received is exceptionally naive. The amount of wise-cracks and beers your team ca…

> The amount of wise-cracks and beers your team can handle after a work day is not the determinate factor in success.

Of course it isn't. But it's a starting point for building a team that can deal with what you describe (a decade-plus long timeline, zero room for failure, etc). If the people responsible are more or less insufferable, progress will be extremely difficult, irrespective of how talented they are.

Re: UK air traffic control meltdown

#287
post #174

Earlier quoted context omitted.

The names have to be entered manually by pilots, if e.g. they change the route. They have to be transmitted over the air by humans. So they must be short ans simple.

Clippy: It looks like you are trying to enter a non unique navigation point, did you mean the one in France or the one in Australia?

if only it were as simple as that - what about unique but easily confusable "human-friendly" identifiers?

As a layman, I'd argue that such efforts would be band-aid and better spent on robust standardization

Re: UK air traffic control meltdown

#289

What I don’t understand in situations like this when thousands of flights are cancelled is how do they catch up? It always seems like flights are at max capacity at all times, at least when I fly. If they cancel 1,000 flights in one day, how do they absorb that extra volume and get everyone where they need to be? Surely a lot of people have their plans permanently cancelled?

Exactly. People's plans get pushed out into the evenings or during the less busy times, absorbed, then forgotten as collateral damage.

Re: UK air traffic control meltdown

#290
post #102

Earlier quoted context omitted.

Well yes because you're describing a system where there are really low stakes and crash recovery is always possible because you can just throw away all your local state. The flip side would be like a database failing to parse some part of its WAL log due to disk corruption and just said, "eh just delete those sections and move on."

Crash the tab and allow all the others to carry on! The problem here is that one individual document failed to parse.

The other “tabs” here are other airplanes in flight, depending on being able to land before they run out of fuel. You don’t just ignore one and move on.
Post reply on HN