I wonder if all of their computer systems have ECC RAM. Kinda doubt it.
A corrupt file led to the FAA ground stoppage – also found in backup system
321–330 of 402 posts
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#322Earlier quoted context omitted.
Looking at the response in the rest of the comment chain I wonder if it's such that he knows but does not accept the change.
I'd never heard of it, and doubt that there's any other explanation than pandering.
> I'd never heard of it,
Now you did.
> doubt that there's any other explanation than pandering.
The airplane doesn't care about the gender or sex of the pilot. There is no pandering in that observation.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#323Earlier quoted context omitted.
> It’s absolutely a corruption issue, in that the government prefers to pay 2-3x Being serious, I wish I could find one of these 2-3x multiple payouts in government. Every time I've looked at anything government related (including direct contractors) the pay is garbage. Usually 15% to 50% of what the private market pays for my skill set.
Worked for a govt contractor years ago and I would imagine the payout from the govt contract was at least 2-3x. There were usually 2-3 layers of people getting paid and I don't think any of them were hurting. Of course no employee was getting a massive payout compared to private. Govt > contractor > sub-contractor > employee was pretty common. I never knew of anyone being a direct contractor to the agency since the c…
It's been several years since I've done contracting work (and for the FAA no less), but that can't really happen to the extent you're implying. The open roles for contract positions specify a pay range based on experience (degrees and/or years of experience). You can have a prime and then a sub contractor, but the amount that each can add for management overhead is limited and depends on the contract vehicle. I want to say it was around 5-10%.
The actual employer can pay the employed contractor whatever they want, but the rates are published and if they underpay too much the employee will be poached by a competitor. Because the rates are public info the employee can look up how much profit their employer is making any time they want.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#324Earlier quoted context omitted.
> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strateg…
We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.
A "simple query". The tech "looks ancient". You're "willing to bet" things. And yet you speak so confidently and derisively about this outage.
It must be nice to sit behind your keyboard and just have all of the answers all day long! Do you have any tips for how to be so omniscient?
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#325Earlier quoted context omitted.
This is a sweeping claim with really nothing to back it up. Are you saying this from an inside knowledge of the FAA, or is this just an opinion? On the surface, the relative safety of air travel and the lack of major stoppages over a span of 22 years seems like a major counter example. You’re making this statement emphatically and authoritatively, though, so I’m curious to understand where that certainty comes from a…
Two examples: 1. The FAA basically handed their risk-management keys over to Boeing when authorizing the 737-MAX, contributing to those deaths ( https://www.newyorker.com/news/our-columnists/how-boeing-and... ) 2. The FAA's pilot medical vetting process, while thorough, is behind the times. There are people who took ADHD medicine in high school that are unable to obtain a medical certificate due to the FAA's overly-s…
The FAA was gutted in the name of deregulation and competition - you might want to ask the GOP what happens when you don't have adequate govt oversight and regulation.
Two planes crashed due to a design flaw - hundreds of people killed and one manufacturer and model forever tarnished like McDonald Douglass and their DC-10.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#326Earlier quoted context omitted.
I hear you. But the flip side is there is often a tendency to massively over complicate things in a way that bakes in valueless complexity and bloat, which is very much the norm in large government systems. I’m practice this attitude is far more common and destructive long term.
I think it's way worse than adding complexity and bloat. There are processes that specifically prevent anyone from understanding or owning the system. The complexity and bloat is a side product of the fact that the work was siloedd, contracted out, and everyone washed their hands off the result. Which also takes exponentially more time and people. Its optimized for inefficiency.
Do you know that each of these systems actually has a defined "owner"? It's part of the FISMA process and every so often these "owners" have to attest that certain security processes are in place and functioning and that no significant changes have been made without a full review... among many other things.
That "owner" is also a federal employee, not a contractor.
https://en.wikipedia.org/wiki/Federal_Information_Security_M...
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#327You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…
The only way to optimize for lowest overall risk is to optimize for speed of change. All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it. Instead, if you can move fast consistently, you can minimize the total downtime.
>All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will)
Eventually, but not as frequently.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#328Earlier quoted context omitted.
I think you're misunderstanding what the "backup" was in this context (in particular as the article is unhelpful by calling it a backup file, which isn't accurate). In this case it is two PRODUCTION systems running concurrently. The primary and the secondary (article calls the "backup"). Primary went down due to corruption, but the identical secondary system couldn't be switched to because the corruption also occurre…
So the backup system failed
No, it's the RAID vs Backup distinction. If you have 2 disks in Raid 1, you have a system that can survive one drive dying (high availability), but you also need a backup system (offline).
It sounds like their high availability system failed but they probably were able to restore from backup (offline)
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#329>In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system – a significant decision, because the reboot can take about 90 minutes, according to the source. How often is the system rebooted? I haven't heard of it happening before and some quick searching didn't find any historical examples. Is it a scheduled event and no flights have departure times while thi…
It sounds more like there are a pair of systems where one operates as the primary and the other as a backup, and these can swap in case of problems with the primary. That's fairly typical in critical infrastructure.
I've done testing on these types of systems in the past (carefully) and the owners will often let you test against the system that's currently the "backup".
So they probably restarted the "primary" after performing a fail-over.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#330You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…
Yes. Breaking things in aviation frequently results in a body count