Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

Post reply on HN
Print view View original

261–270 of 459 posts

Re: UK air traffic control meltdown

#261

Post by drachir91 »

burntwater wrote: post #242
drachir91 wrote: post #207

What ticked me is that when the primary system threw in the towel, an EXACT SAME system took over and ran the exact same code on the exact same data as the primary. I know that with code and algorithms it's not always the case but even then you know what doing the same thing over and over expecting different results defines... Yes, it can be argued that the software should've had more graceful failure modes and this…

I'm wondering if the backup system could have a delayed queue; say, 30 seconds behind. If the primary fails, and exactly 30 seconds later the secondary system fails, you have reasonable assurance that it was queue input that caused the failure. Rollback to the last successful queue input, skip and flag the suspect input, and see if the next input is successful.

This looks to me like it could work, but would need a ready force of technicians always expecting something like that so they can troubleshoot it in a timely manner.

Re: UK air traffic control meltdown

#262

Post by sublimefire »

crabbone wrote: post #256
samus wrote: post #155

Earlier quoted context omitted.

That reasoning is fine, but it rather seems that the programmers triggered this catastrophic "stop the world" error because they were not thorough enough considering all scenarios. As TA expounds, it seems that neither formal methods nor fuzzing were used, which would have gone a long way flushing out such errors.

Neither formal methods nor fuzzing would've helped if the programmer didn't know that input can repeat. Maybe they just didn't read the paragraph in whatever document describes how this should work and didn't know about it. I didn't have to implement flight control software, but I had to write some stuff described by MIFID. It's a job from hell, if you take it seriously. It's a series of normative documents that expl…

I've only heard from people engineering systems for aerospace industry and we're speaking hundreds of pages of api documentation. It is very complex so equally the chances of a human error are higher.

Re: UK air traffic control meltdown

#264

Post by CamelCaseName »

jliptzin wrote: post #254

What I don’t understand in situations like this when thousands of flights are cancelled is how do they catch up? It always seems like flights are at max capacity at all times, at least when I fly. If they cancel 1,000 flights in one day, how do they absorb that extra volume and get everyone where they need to be? Surely a lot of people have their plans permanently cancelled?

There's always some empty capacity, whether it's non-rev tickets for flight crew and their families which are lower priority than paying customers or people who miss their flights.

I had a cancelled flight recently and they booked people two weeks out because every flight from that day onward was full or nearly full. I showed up the next morning and was able to board the next flight because exactly one person had scanned in their boarding pass (was present at the airport) but did not show up for whatever reason to the airplane.

Beyond that, people just make alternate plans, whether it's taking a bus or taxi home, traveling elsewhere, picking another airline, anything is possible.

Re: UK air traffic control meltdown

#265

Post by ftxbro »

NBJack wrote: post #255
ftxbro wrote: post #251

Earlier quoted context omitted.

> Long story: because changing identifiers is a considerable refactoring is this what refactoring means

Yes. It would cascade into: Changes in how ATCs operate Changes in how pilots operate Changes in how airplanes receive these instructions (including the flight software itself, safety systems, etc.) Changes in how airplanes are tested Changes in how pilots are trained Etc. In this case, the refactoring requires changes to hardware, software, training, manufacturing, and humans.

does refactoring mean literally any non-local change even just like changing a variable name, or does it usually mean some kind of structural or architectural non-local change

Re: UK air traffic control meltdown

#266

Post by Scarblac »

slt2021 wrote: post #66
cratermoon wrote: post #5

Earlier quoted context omitted.

> why could the system not put the failed flight plan in a queue Because it doesn't look at the data as a "flight plan" consisting of "way points" with "segments" along a "route" that has any internal self-consistency. It's a bag of strings and numbers that's parsed and the result passed along, if parsing is successful. If not, give up. In this case fail the entire system and take it out of production. Airline indust…

good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest. seems like poor engineering

This flight plan was correct though, if there was some validation like that then it should have passed.

The code that crashed had a bug, it couldn't deal with all valid data.

Re: UK air traffic control meltdown

#267

Post by thedrbrian »

jliptzin wrote: post #254

What I don’t understand in situations like this when thousands of flights are cancelled is how do they catch up? It always seems like flights are at max capacity at all times, at least when I fly. If they cancel 1,000 flights in one day, how do they absorb that extra volume and get everyone where they need to be? Surely a lot of people have their plans permanently cancelled?

You don't.

I work in logistics for a FMCG company and sometimes our main producer goes down and we run out of certain types of stock. We send as much out as we can and cancel the rest.

If they really want the stock the customers can rebook an order for tomorrow because they aren't getting it today. And we just start adding extra stock to each delivery.

It's the best of a bad situation.

We don't have the money to have extra trucks and very perishable stock laying about and I know the airlines don't pay 300 grand a month to lease a 737 just to have it sat about doing nothing. There's very little slack.

Re: UK air traffic control meltdown

#268

Post by bsder »

epanchin wrote: post #215
sam0x17 wrote: post #163

Earlier quoted context omitted.

Why on earth do they not have GUIDs for these navigation points if the names are not globally unique and inter-region routes are commonplace?

What three words would be a better solution than a guid, as transmittable over radio.

WTW is a proprietary system that should never be used:

https://www.walklakes.co.uk/opus64534.html

The biggest fault (besides being proprietary) is that you must be online in order to use WTW. The times that you might need WTW are ALSO the times you are most likely to be unable to be online.

Re: UK air traffic control meltdown

#269

Post by deadfish »

NBJack wrote: post #255
ftxbro wrote: post #251

Earlier quoted context omitted.

> Long story: because changing identifiers is a considerable refactoring is this what refactoring means

Yes. It would cascade into: Changes in how ATCs operate Changes in how pilots operate Changes in how airplanes receive these instructions (including the flight software itself, safety systems, etc.) Changes in how airplanes are tested Changes in how pilots are trained Etc. In this case, the refactoring requires changes to hardware, software, training, manufacturing, and humans.

Pretty sure that is still not the meaning of refactoring. As I understand it refactoring should mean no changes to the external interface but changes to how it is implemented internally.

Re: UK air traffic control meltdown

#270

Post by tppiotrowski »

reactordev wrote: post #28

So they forgot to "geographically disparate" fence their queries. Having built a flight navigation system before, I know this bug. I've seen this bug. I've followed the spec to include a geofence to avoid this bug.

ICAO standard effective from 1978 to only duplicate identifiers if more than 600 nmi (690 mi; 1,100 km) apart
Post reply on HN