Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

111–120 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#111

Post by techie128 »

lopkeny12ko wrote: post #49
tigerBL00D wrote: post #35

Earlier quoted context omitted.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

> If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

There are databases that maintain redundant copies and can tolerate disk / replica failure. e.g. Cassandra.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#112

Post by stingraycharles »

kortilla wrote: post #83
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

I see this a lot with people are experts in real time operating systems environments, particularly in aviation/space stuff (maybe because that’s where I worked for a while). They have excellent intuition around making things redundant to single pieces of hardware failing but don’t really grok making stuff resilient to wider failures. Anything involving transaction logs, rollbacks, and plain old backups take a backsea…

> They have excellent intuition around making things redundant to single pieces of hardware failing but don’t really grok making stuff resilient to wider failures.

I always feel like making single components redundant is a fairly well-defined process -- generally speaking, the mechanisms are the same (1+ redundant components, failover, STONITH, etc), where making things resilient on a higher level is not as well-defined, and often requires bespoke solutions to each unique situation.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#113

Post by jacquesm »

dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

It's risk avoidance to the point that that avoidance leads to new kinds of risks. The whole idea that you can architect yourself out of failure modes to the point that you no longer need to make backups is one that I see every other week or so and the number of companies out there that believes that because they have redundancies they don't need backups any more is staggering.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#114

Post by simonh »

dmix wrote: post #90
jrockway wrote: post #80

Earlier quoted context omitted.

What's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software. Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. N…

This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you kee…

It’s a fundamental limitation of identical redundant systems that they have vulnerability to some of the same threats, particularly bad inputs and capacity issues. It’s important to understand it’s only giving you physical redundancy, such as if one data centre goes down. But the same software bugs, the same bad input data, even the same memory overruns are likely to hit both systems.

It’s not bad design, it’s just you have to understand what resiliency you have and plan against each of various such threats according to your risk appetite.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#115

Post by SSLy »

tyingq wrote: post #26
rawgabbit wrote: post #3

I wish to know what kind of system NOTAM runs on. An IBM 360 with tape backup?

You can see references to JMS, Solace, and Weblogic in various docs, like: https://www.faa.gov/air_traffic/technology/swim/users_forum/... https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%... So probably Linux, though perhaps something else.

Could be a Solaris.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#117

Post by defrost »

SSLy wrote: post #115
tyingq wrote: post #26

Earlier quoted context omitted.

You can see references to JMS, Solace, and Weblogic in various docs, like: https://www.faa.gov/air_traffic/technology/swim/users_forum/... https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%... So probably Linux, though perhaps something else.

Could be a Solaris.

Given the vintage pedigree BSD like SunOS is an even better candidate.

Once tuned for processing they just ran - rip out a network cable and stuff it back in again, not a hiccup.

Many of the time saw no benefit in porting forward to Slow Loris.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#118

Post by f1refly »

pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

TDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#119

Post by cbrogrammer »

f1refly wrote: post #118
pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

TDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.

TDD helps close to 0 % with fragility and only exists to help development

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#120

Post by nix23 »

chx wrote: post #93

90 minutes boot time... I have a feeling the summarized age of the hard drives in that system exceeds the age of the United States itself (which is 247 years). When fsck was introduced in 4BSD in 1980, it checked every filesystem on boot because it had no better idea. If this thing is, say, forty years old then that's exactly the right age for this...

Maybe you haven't seen big servers, but some blade-servers need >15 minutes just for the hardware check, same with big switches and san's.
Post reply on HN