Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

301–310 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#301

Post by eckza »

justinsaccount wrote: post #296
alexb_ wrote: post #271

Earlier quoted context omitted.

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

People like to think that the alternative to "move fast and break things" is "move slowly and not break things" but it's not, it's "move slowly, break things anyway, then take days to resolve the problem because you never learned how to move fast".

That's just not true.

Moving faster does not mean moving more effectively. Frantic activity should not be mistaken for progress.

I'm OK with the FAA taking "days to resolve the problem" if it means that nobody dies.

For what it's worth, my team and I are partial to "move carefully, and tend things".

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#302

Post by haswell »

njovin wrote: post #287
haswell wrote: post #279

Earlier quoted context omitted.

This is a sweeping claim with really nothing to back it up. Are you saying this from an inside knowledge of the FAA, or is this just an opinion? On the surface, the relative safety of air travel and the lack of major stoppages over a span of 22 years seems like a major counter example. You’re making this statement emphatically and authoritatively, though, so I’m curious to understand where that certainty comes from a…

Two examples: 1. The FAA basically handed their risk-management keys over to Boeing when authorizing the 737-MAX, contributing to those deaths ( https://www.newyorker.com/news/our-columnists/how-boeing-and... ) 2. The FAA's pilot medical vetting process, while thorough, is behind the times. There are people who took ADHD medicine in high school that are unable to obtain a medical certificate due to the FAA's overly-s…

> 1. The FAA basically handed their risk-management keys over to Boeing when authorizing the 737-MAX, contributing to those deaths

Couldn’t this also be interpreted as: when the FAA holds the keys, disasters like the 737-MAX tend not to happen? Obviously this raises questions about how that decision came about in the first place, but as an example, it seems counterproductive to your point, i.e. evidence that shifting away from some long standing policies directly led to harm, implying the original policies might have been better ones.

In a thread that seems eager to move fast and break things, this seems like a big problem, and would seem to indicate the need for a return to founding principles, not the opposite.

> 2. The FAA's pilot medical vetting process

This is an interesting one for sure, but also seems like an incredibly complex issue. Have there been studies about the safety of operating machinery while on those drugs that would obviate the need for a policy change?

The potential risk averted by such a policy would need to be weighed against the negative impacts of the 2nd order undesirable behaviors - obviously it’s bad that the policy discourages much needed mental health support, but how bad this is depends entirely on how effective the initial screening process is.

I’m not saying these mental health policies shouldn’t be changed, but neither do they seem to have obvious or measurably better alternatives at the moment.

And taken in the context of the original claim - that people are grossly misunderstanding the FAA and all of this is theater - they seem like weak examples to use as evidence of broad organizational failure.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#303

Post by phaedrus »

JCM9 wrote: post #200
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor…

One common problem is building new things with tech that is already obsolete. Given the choice between new and old, the old is perceived as more tried and tested. Granted it's sometimes difficult to distinguish "new and going to last" from "new and shiny", but when it comes to software choosing to go with "old and tested" can be counter to cyber security since the old software is no longer updated.

Of course you also have "old but still developed, and likely to be supported in near-perpetuity". Things like SQLite fall in this category. Unfortunately there's this other problem of management being inexplicably down on anything open-source. I don't know whether they've been exposed to too much FUD from vendors selling proprietary solutions, or just that the idea that security through obscurity is no security at all has failed to reach them.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#304

Post by bastardoperator »

malfist wrote: post #228
gnarbarian wrote: post #224

Earlier quoted context omitted.

demonstrably the policies employed by the FAA do not effectively avert risk (in software). It's more akin to risk aversion theater.

Which is why we've not had a commercial plane crash since.......2009? Seems FAA is pretty effective at reducing risk.

I haven't had a car crash, should I also thank the FAA?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#305

Post by phaedrus »

simonh wrote: post #114
dmix wrote: post #90

Earlier quoted context omitted.

This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you kee…

It’s a fundamental limitation of identical redundant systems that they have vulnerability to some of the same threats, particularly bad inputs and capacity issues. It’s important to understand it’s only giving you physical redundancy, such as if one data centre goes down. But the same software bugs, the same bad input data, even the same memory overruns are likely to hit both systems. It’s not bad design, it’s just y…

It's very much like the advice that mirror RAID is not a backup solution. That seems to be almost literally what went wrong here.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#307

Post by phaedrus »

Someone1234 wrote: post #215
VLM wrote: post #206

"The source said the NOTAM system is an example of aging infrastructure due for an overhaul." The concept of "lint" a file and the concept of verifying a backup are truly ancient and coincidentally are also completely absent from the description of the problem. People that felt no need to do either 20 years ago are certainly not going to start doing it today, especially when the inevitable system failure in the dista…

I think you're misunderstanding what the "backup" was in this context (in particular as the article is unhelpful by calling it a backup file, which isn't accurate). In this case it is two PRODUCTION systems running concurrently. The primary and the secondary (article calls the "backup"). Primary went down due to corruption, but the identical secondary system couldn't be switched to because the corruption also occurre…

Perhaps they should have called it a "tandem" system or hot-standby.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#308

Post by thrashh »

bastardoperator wrote: post #298
haswell wrote: post #290

Earlier quoted context omitted.

> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strateg…

We can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.

We found the person who intimately knows how FAA’s system is engineered and who also builds perfect systems

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#309

Post by shadowgovt »

lioeters wrote: post #236
nix23 wrote: post #125

Earlier quoted context omitted.

>They're so rare, They are not rare, just because you don't detect them (consumer hardware) means nothing. >>study done by IBM in the 1990’s that referenced 1 cosmic ray bit flip per 256MB of memory per month https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...

One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. I kinda wish I hadn't learned this fact, how fragile everything is. On the other hand, it makes me appreciate the importance of error tolerance and recovery. > Cosmic ray flux depends on altitude. Computers operated on top of mountains experience an order of magnitude higher rate of soft errors compared to…

The reason we all now wear our seatbelts through the entire flight is there was a cross-Pacific flight that had a sudden altitude drop where several people were injured from rebounding off the ceiling. When the news reported on it, they reported the cause was believed to have been an undetected massive downdraft that shoved the plane down.

I lost the thread on that story before the investigation concluded the actual cause was autopilot error. It is, in some ways, more comforting to me to know that the issue wasn't novel atmospheric phenomena, but instead relatively-mundane cosmic radiation flipping one packet of data from the sensors to the avionics that the avionics lacked sufficient redundancy to detect or discard. As a result, the autopilot believed the plane had suddenly pitched 90 degrees and drastically corrected to escape stall.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#310

Post by czbond »

I don't believe it. A system that important would have significant scrutiny on its components even by a legacy organization.

Maybe it was a corrupted HACKED filed, I'd believe that. After all, China isn't too happy with our games about Taiwan.

Post reply on HN