Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

251–260 of 459 posts

Re: UK air traffic control meltdown

#251

Post by ftxbro »

amoerie wrote: post #170
sam0x17 wrote: post #163

Earlier quoted context omitted.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

Long story: because changing identifiers is a considerable refactoring, and it takes coordination with multiple worldwide distributed partners to transition safely from the old to the new system, all to avoid a hypothetical issue some software engineer came up with Short story: money. It costs money to do things well.

> Long story: because changing identifiers is a considerable refactoring

is this what refactoring means

Re: UK air traffic control meltdown

#252

Post by tristor »

rglover wrote: post #229
count wrote: post #164

Earlier quoted context omitted.

Ah yes, migration through sheer force of will.

In some sense, yes. Notice that most of the responses to what I've said are immediately negative or dismissive of the idea. If that's the starting point (bad mindset), of course nothing gets fixed and you land where we are today. My initial approach would be to weed out anyone with that point of view before any work took place (the "not HR friendly" part being to be purposefully exclusionary). The only way a problem…

Anyone who has worked with me knows that I have no issue coming in like a wrecking ball in order to make things happen, when necessary. I've also been involved in some of these migration projects. I think your take on the complexity of these projects (and I do mean inherent complexity, not incidental complexity) and the responses you've received is exceptionally naive.

The amount of wise-cracks and beers your team can handle after a work day is not the determinate factor in success. /Most/ of these organizations /want/ to migrate these systems to something better. There is political will and budget to do so, these are still inglorious multi-decade slogs which cannot fail, ever, because failure means people die. No amount of attitude will change that.

Re: UK air traffic control meltdown

#253

Post by crabbone »

hn_throwaway_99 wrote: post #168
adrianmonk wrote: post #131

Earlier quoted context omitted.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

I agree with the general sentiment "if you see an unexpected error, STOP", but I don't really think that applies here. That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processi…

> is essentially totally independent

They physically cannot be independent. The system works on an assumption that the flight was accepted and is valid, but it cannot place it. What if it accidentally schedules another flight in the same time and place?

Re: UK air traffic control meltdown

#254

Post by jliptzin »

What I don’t understand in situations like this when thousands of flights are cancelled is how do they catch up? It always seems like flights are at max capacity at all times, at least when I fly. If they cancel 1,000 flights in one day, how do they absorb that extra volume and get everyone where they need to be? Surely a lot of people have their plans permanently cancelled?

Re: UK air traffic control meltdown

#255

Post by NBJack »

ftxbro wrote: post #251
amoerie wrote: post #170

Earlier quoted context omitted.

Long story: because changing identifiers is a considerable refactoring, and it takes coordination with multiple worldwide distributed partners to transition safely from the old to the new system, all to avoid a hypothetical issue some software engineer came up with Short story: money. It costs money to do things well.

> Long story: because changing identifiers is a considerable refactoring is this what refactoring means

Yes. It would cascade into:

Changes in how ATCs operate

Changes in how pilots operate

Changes in how airplanes receive these instructions (including the flight software itself, safety systems, etc.)

Changes in how airplanes are tested

Changes in how pilots are trained

Etc. In this case, the refactoring requires changes to hardware, software, training, manufacturing, and humans.

Re: UK air traffic control meltdown

#256

Post by crabbone »

samus wrote: post #155
adrianmonk wrote: post #131

Earlier quoted context omitted.

Because the code classified it as a "this should never happen!" error, and then it happened. The code didn't classify it as a "flight plan has bad data" error or a "flight plan data is OK but we don't support it yet" error. If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have co…

That reasoning is fine, but it rather seems that the programmers triggered this catastrophic "stop the world" error because they were not thorough enough considering all scenarios. As TA expounds, it seems that neither formal methods nor fuzzing were used, which would have gone a long way flushing out such errors.

Neither formal methods nor fuzzing would've helped if the programmer didn't know that input can repeat. Maybe they just didn't read the paragraph in whatever document describes how this should work and didn't know about it.

I didn't have to implement flight control software, but I had to write some stuff described by MIFID. It's a job from hell, if you take it seriously. It's a series of normative documents that explains how banks have to interact with each other which were published quicker than they could've been implemented (and therefore the date they had to take effect was rescheduled several times).

These documents aren't structured to answer every question a programmer might have. Sometimes the "interesting" information is close together. Sometimes you need to guess the keyword you need to search for to discover all the "interesting" parts... and it could be thousands of pages long.

Re: UK air traffic control meltdown

#257

Post by drachir91 »

throwaway894345 wrote: post #216
drachir91 wrote: post #207

What ticked me is that when the primary system threw in the towel, an EXACT SAME system took over and ran the exact same code on the exact same data as the primary. I know that with code and algorithms it's not always the case but even then you know what doing the same thing over and over expecting different results defines... Yes, it can be argued that the software should've had more graceful failure modes and this…

> Aside from cost. I don't care about anyones cost-cutting incentives in mission-critical systems. Sorry capitalism... Capitalism is happy to have redundancy in mission critical systems all the time. Why would it care here?

I don't know but in recent years I'm increasingly seeing mission critical systems having only token or "apparent" rendundancies instead of real ones, and couldn't find any other rationale than cost savings and shareholder bottom lines. I'm not saying that capitalism = bad, it's mostly better than the alternatives, but just like its most direct competitor, it suffers from bad implementations across the world and unbounded human greed.

A recent and very "in the face" example, also from the air travel industry would be the B737 Max and its AoA sensors. There were two, for two flight computers, but MCAS only used 1 flight computer and 1 AoA sensor, despite the already existing crosslinks between the flight computers and the sensors...

Pofit maxing first with the "no need for a new type rating for the pilots", then cost-cutting first in aeronautical engineering (solving an airframe design problem with software, plus designing a flight envelope protection system that can overpower the human pilots).

Then cost-cutting in software engineering and QC, rushing out software made by (probably) inexperienced in the field engineers and failing to properly test it and ensure that it had the needed redundancy.

Re: UK air traffic control meltdown

#258

Post by crabbone »

pmontra wrote: post #186
raverbashing wrote: post #154

Earlier quoted context omitted.

And that's why I never (or very rarely) put "this should never happen" exceptions anymore in my code Because you eventually figure out that, yes, it does happen

A customer of mine is adamant in their resolve to log errors, retry a few times, give up and go on with the next item to process. That would have grounded only the plane with the flight plan that the UK system could not process. Still a bug but with less effects to all the continent, because planes that could not get inside or outside the UK could not fly and that affected all of Europe and possibly more.

> That would have grounded only the plane with the flight plan that the UK system could not process.

By the looks of it, it was few hours in the air by the time the system had a breakdown. Considering it didn't know what the problem was, it seems appropriate that it shut down. No planes collided, so the worst didn't happen.

Re: UK air traffic control meltdown

#259

Post by sublimefire »

dboreham wrote: post #200
codeulike wrote: post #153

This is a great post. My reading of it: - waypoint names used around the world are not unique - as a sortof cludge, "In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced." - but still you might get the same waypoint name used twice in a route to mean different places - the software was not written with that possibilty in mind - route did not compute…

"software supplier"??? Why on God's green earth isn't someone familiar with the code on 7/24 pager duty for a system with this level of mission criticality?

I think there is a bit of ignorance about how software is sold in some cases. This is not just some windows or browser application that was sold but it also contained the staff training with a help to procure hardware to run that software and maybe even more. Such systems get closed off from the outside without a way to send telemetry to the public internet (I've seen this before, it is bizarre and hard to deal with). The contract would have some clauses that deal with such situations where you will always have someone on call as the last line of defense if a critical issue happens. Otherwise, the trained teams should have been able to deal with it but could not.

Re: UK air traffic control meltdown

#260

Post by drachir91 »

crabbone wrote: post #236
drachir91 wrote: post #207

What ticked me is that when the primary system threw in the towel, an EXACT SAME system took over and ran the exact same code on the exact same data as the primary. I know that with code and algorithms it's not always the case but even then you know what doing the same thing over and over expecting different results defines... Yes, it can be argued that the software should've had more graceful failure modes and this…

> an EXACT SAME system took over and ran the exact same code Did you ever work with HA systems? Because this is how they work. It's two copies of the same system intended for the cases when eg. hardware fails, or network partitioning happens etc.

No, I do not. But HA systems work like that because hardware or network failure is what they are designed to guard against, not a latent bug in the software logic. If there's a software bug, both systems will exhibit the same behavior, so HA fails there.
Post reply on HN