Bunch of Node.js programmers lecturing the FAA on reliability is hilarious. Your framework is out of date, go use whatever the latest js framework published today is.
I'm sorry you can't fly today because the NOTAM system sent NaN alerts.
A corrupt file led to the FAA ground stoppage – also found in backup system
311–320 of 402 posts
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#312You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…
The only way to optimize for lowest overall risk is to optimize for speed of change.
All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it.
Instead, if you can move fast consistently, you can minimize the total downtime.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#313> NOTAMs (Notice to Air Missions) OT, but didn't NOTAM use to mean Notice to Airmen? It's still that on the ICAO website: https://www.icao.int/safety/istars/pages/notams.aspx Did it change, or is this a CNN initiative? (And how does one notify a mission?)
The US changed it. AFAIK, the icao definition has not changed, though I guess it could in the future.
Given that the US often does these kinds of things first, I've often wondered if it is a US-centric way of thinking?
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#314Earlier quoted context omitted.
> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strateg…
We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.
This has nothing to do with excuses - I’m challenging the assertion that “because something bad happened, they must not have any mitigations in place at all”.
This seems like a bad case of binary thinking, and my point was that the occurrence of an incident like this is not sufficient to support that claim. It’s just as likely that an ancient process that wasn’t accounted for somewhere in the architecture broke down, and this is how it manifested.
Clearly improvements are needed, as is always the case after an outage. That doesn’t justify wild speculation.
Anecdote time: I once worked for a large financial institution that makes money when people swipe their credit cards. The system that authorizes purchases is ancient, battle tested, and undergoes minimal change because the cost of an outage could be measured in the millions of $ per minute.
Every change was scrutinized, reviewed by multiple groups, discussed with executives, and tested thoroughly. The same system underwent regular DR testing that involved quite a lot of involvement from all related teams.
So the day it went down, it was obviously a big deal, and raised all of the natural questions about how such a thing could occur.
Turns out it had an unknown transitive dependency on an internal server - a server that had not been rebooted in literally a decade. When that server was rebooted (I think it was a security group insisting it needed patches despite some strong reasons to avoid that when considering the architecture), some of the services never came back up, and everyone quickly learned that a very old change that predated almost everyone there established this unknown dependency.
The point of this story is really about the unknowability of sufficiently complex legacy enterprise systems.
All of the right processes and procedures won’t necessarily account for that seemingly inconsequential RPC call to an internal system implemented by a grizzled dev shortly before his retirement.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#315Earlier quoted context omitted.
> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strateg…
We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.
We can make shit up and pretend to be experts and criticize things we know absolutely nothing about all day long, too.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#316Earlier quoted context omitted.
> They should be testing their databases before backing them up blindly. Oh you mean they should be testing/validating the generated backup db file before replicating it to long-term archive ...
Way back when use cases were a thing, I used to chide people for saying that Backup was a use case. No, Restore is a use case. (Replace "use case" with "requirement" or "user story"...)
A corollary to this would be: “Backups are worthless. Restores are priceless.”
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#317In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not c…
Related: http://www.taobackup.com/
Thanks! I haven't seen this before, though I have heard these 7 general principles. My personal strategy is broadly:
1. daily encrypted restic backups to my local NAS, running ZFS raidz2
2. weekly rsyncs of the backup directory to an offsite machine
3. yearly full offsites of the entire NAS
I test my backups frequently if only because I have some odd persistent bug that eats my zsh history file about once a month. I now have a "recover-latest-history" script that pulls that file back down from the latest restic backup. I have just completed an offsite and exchanged it for last year's, so I can test that now too!
Rate my tao :)
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#318Earlier quoted context omitted.
You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.
People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".
You act like "moving fast" is all you need to know to "move fast". As if it's simply the skill of making time move faster, and you don't need any other skills than that, because once you've broken the laws of physics and changed the speed of time, everything just works faster without any differences or consequences. Do you watch a lot of Superhero movies?
You should work smarter, not harder. Just turn up your smart knob. But why didn't you ever think of that before? Probably because you had your smart knob turned all the way down.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#319Earlier quoted context omitted.
People who want a job like that should just be on UBI instead. Then at least we'd have systems that could change to meet the needs of their users in a timely way.
UBI will never pay what a government job pays in purchasing power; that part is just math I think.
I don't think that is unreasonable to expect that government pay in a world where the government is a welfare program with a governing hobby might have less purchasing power than UBI in a world where we prioritize effective governance over beaurocracy. It's not a zero sum game.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#320Earlier quoted context omitted.
There’s a reason for that. Freight has priority because the freight companies actually own the rails, and actually use them, and it’s a significant contributor to the US economy. Amtrak is an unwelcome guest that they are required by law to put up with. If they wanted better service, Amtrak could pay for it, except the big cross country routes are already crazy unprofitable. The actual sane solution is to shut down t…
Your comment gives the impression that Amtrak has no business operating on the tracks they do. The reason the freight companies own those rails is primarily from the land grants made my the federal government which came with obligations, such as providing passenger service. Amtrak has trackage rights because the freight companies wanted to divest their passenger rail operations. Maybe an unwelcome guest, but essentia…
Yeah. They gave railroads the initial land — in 1872. There’s been a lot that’s happened since then, like the bankruptcy and near-complete collapse of functional passenger rail transportation in most areas in the 1970s-1980s, as it faced stunning new competition from cars and planes.
But the argument of “is this justified given the history?!” should take a back seat; certainly Congress is able to force the industry’s hand whether or not it’s justified. The first question should be whether it’s a good idea in the first place, since rail is doing useful things for the US economy, which would ultimately shoulder more costs for it than the railroads themselves as a business.
Damaging the supply chain and raising prices across the economy while putting more trucks on the taxpayer-funded roads emitting more carbon dioxide is a steep price. Incremental improvements to the reliability of seldom-used cross-country routes through the sparsely inhabited West at speeds of about 60mph aren’t worth that price.