31–40 of 459 posts
Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.
And why could the system not put the failed flight plan in a queue for human review and just keep on working for the rest of the flights? I think the lack of that “feature” is what I find so boggling.
Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed.
For these kinds of things the post mortem and remediation have to kinda take as given that eventually a not predictable in advance unhandled unknown error will occur and then work on how it could be handled better. Because of course the solution to a bug is to fix the bug, but the issue and the reason for the meltdown is a DR plan that couldn't be implemented in a reasonable timeframe. I don't care what programming practices, what style, what language, what tooling. Something of a similar caliber will happen again eventually with probability 1 even with the best coders.
Earlier quoted context omitted.
As is tradition =) (We actaully have no major issues with the French at least in my generation, its all just good fun)
It’s all good. We still take cheap-shots at English food and English women ;) Edit: I lived in London for 3 years. I miss it every day.
Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…
Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…
They bought a software from a third party and treat it as a "black box". There are few known ways that the software fails, and the local team has instructions on how to fix it. But if it fails in an unexpected way, good luck, it's impossible for the local team to identify and fix the problem without the vendor.
The reason it took so much was they realized too late that they need to call the vendor.
Probably you have to blame managers rather than engineers in the support team.
Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…
> What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input...) To collect money on support contracts, I suspect.
This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability. Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.
Software has bugs, that's not really the damning part... The damning part is that in four hours and two levels of support teams, there was noone who actually knew anything about how the system worked who could remove the problematic flight plan so that the rest of the system could continue operating! What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input..…
Earlier quoted context omitted.
It’s all good. We still take cheap-shots at English food and English women ;) Edit: I lived in London for 3 years. I miss it every day.
I wouldn't worry about it, we take cheap shots at French food and French people too ;)