Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

281–290 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#281

Post by nix23 »

lioeters wrote: post #236
nix23 wrote: post #125

Earlier quoted context omitted.

>They're so rare, They are not rare, just because you don't detect them (consumer hardware) means nothing. >>study done by IBM in the 1990’s that referenced 1 cosmic ray bit flip per 256MB of memory per month https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...

One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. I kinda wish I hadn't learned this fact, how fragile everything is. On the other hand, it makes me appreciate the importance of error tolerance and recovery. > Cosmic ray flux depends on altitude. Computers operated on top of mountains experience an order of magnitude higher rate of soft errors compared to…

>One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes.

Yes thats why you have max altitude for normal servers/hardware and it's also more disastrous on modern HW -> smaller transistors.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#282

Post by bastardoperator »

alexb_ wrote: post #271
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#283

Post by tyingq »

huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

>The FAA handled this outage appropriately and in a timely manner

It's still not working as designed, right now. The workarounds in place are fine, but were made up on the fly. Certainly there's room for improvement.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#284

Post by phaedrus »

dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

This reminds me of a particular hardware system I'm familiar with whose design specified "dual power supplies". However late in the acceptance process it was discovered that the condition of one power supply up and one power supply down causes the system to lock up. (My guess would be a phantom current path between the powered and un-powered halves causing an unintended circuit state. Or perhaps just a software bug; maybe the one-supply-down notification code path was never tested.)

The vendor simply changed the procedures to say the user must use a two-fingers procedure to simultaneously flip both power supply switches on or both off at once. It bothers me that we still don't know WHY the original problem happened. How do we know there isn't electrical damage occurring during the brief period between the two switch contacts (since no human can do that perfectly)? If it's a software problem, what is the most time one power supply can be up and one down before the bug is triggered? That's not been characterized, to my knowledge.

But in the context of this thread, what's relevant is to avoid addressing the problem the vendor changed the meaning of "dual power supplies" from an OR condition to an AND condition! They met the letter of the spec while completely violating the spirit of the requirement.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#285

Post by gnarbarian »

nonethewiser wrote: post #256
JCM9 wrote: post #237

Earlier quoted context omitted.

I hear you. But the flip side is there is often a tendency to massively over complicate things in a way that bakes in valueless complexity and bloat, which is very much the norm in large government systems. I’m practice this attitude is far more common and destructive long term.

I think it's way worse than adding complexity and bloat. There are processes that specifically prevent anyone from understanding or owning the system. The complexity and bloat is a side product of the fact that the work was siloedd, contracted out, and everyone washed their hands off the result. Which also takes exponentially more time and people. Its optimized for inefficiency.

this is extremely accurate. it's like a giant rube Goldberg machine that takes both good code and garbage as input and produces a tremendous amount of garbage that nobody can understand once it comes out the other end.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#287

Post by njovin »

haswell wrote: post #279
JCM9 wrote: post #231

Earlier quoted context omitted.

This demonstrates a gross misunderstanding of how the FAA actually operates in its efforts to address “risk.” It’s far more theatre via bureaucracy and paperwork than actual proven engineering efforts that demonstrably reduce risk.

This is a sweeping claim with really nothing to back it up. Are you saying this from an inside knowledge of the FAA, or is this just an opinion? On the surface, the relative safety of air travel and the lack of major stoppages over a span of 22 years seems like a major counter example. You’re making this statement emphatically and authoritatively, though, so I’m curious to understand where that certainty comes from a…

Two examples:

1. The FAA basically handed their risk-management keys over to Boeing when authorizing the 737-MAX, contributing to those deaths (https://www.newyorker.com/news/our-columnists/how-boeing-and...)

2. The FAA's pilot medical vetting process, while thorough, is behind the times. There are people who took ADHD medicine in high school that are unable to obtain a medical certificate due to the FAA's overly-strict policies on prescription drugs. There are current pilots with serious mental issues who are afraid to see a doctor about them due to fear of losing their medical license (https://www.flyingmag.com/why-pilots-dont-want-to-talk-about...).

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#288

Post by marmetio »

huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

I guess this is the comment chain where we address the room?

I'm seeing a lot of misuse of the word "risk". The FAA prioritizes safety over mission. The mishap that resulted in downtime affected the mission. The common-cause failure of the backup system affected the mission. That is not evidence that they're bad at managing safety risk.

Given that there was an article published at the time about removing the dissimilar backup, it's probable that they explicitly accepted this mission risk.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#289

Post by gnarbarian »

CamperBob2 wrote: post #258
Someone1234 wrote: post #214

Earlier quoted context omitted.

"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code. Obviously the NOT…

What makes this a hard problem, exactly? It has to work no matter what, but it also doesn't have to do very much.

I've worked directly with this system a number of times. it's basically a pub sub rss feed optimized for low latency and molested by bureaucrats for decades

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#290

Post by haswell »

bastardoperator wrote: post #282
alexb_ wrote: post #271

Earlier quoted context omitted.

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

If the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.

It is possible to have all of those mitigations in place and still experience a failure like this.

Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure.

A DR strategy is just that - a strategy.

A failure of this sort is not an automatic implication that those things do not exist, just that they failed in this particular case.

I would find it incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident if none of those things were in place.

They’d be either incredibly lucky, or incredibly competent, and if they are the latter, they would not operate without such mitigations in place.

It seems far more believable that an organization of the FAA’s age and complexity missed something along the way.

Post reply on HN