Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

81–90 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#81

Post by defrost »

shemnon42 wrote: post #79
GalenErso wrote: post #11

Earlier quoted context omitted.

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.

Cosmic rays are more common than you think. Google's early infrastructure was impacted by a supernova (because their nodes were so cheap). But something like NOTAM can handle these single bit flips without a problem. https://www.newyorker.com/magazine/2018/12/10/the-friendship... File this under "early Google's infrastructure was a low grade cosmic ray detector."

You can expect 250 or so cosmic ray events per second in a 42 litre sodium iodide crystal pack at 100m above sea level.

Source: 10 years airbourne geophysics, radiometric calibrations.

Addendum:

In-flight upset 154 km west of Learmonth, WA 7 October 2008 VH-QPA Airbus A330-303 [1] was a probable (but uncertain) example of cosmic ray events causing multiple spikes in one of three air data inertial reference units (ADIRUs) that also went on to cause a failure mode of the "best of three" reporting system leading to a pitch down in which [2]

> 110 of the 303 passengers and nine of the 12 crew members were injured; 12 of the occupants were seriously injured and another 39 received hospital medical treatment.

HOWEVER .. despite 250 events per second in a 42 litre volume, it took 128 million hours of unit operation to see a failure mode.

It's a lot of billiard balls going through a lot of space and a high bar for "something bad" ( just the right bit flip ) to happen.

[1] https://www.atsb.gov.au/sites/default/files/media/3532398/ao...

[2] https://www.atsb.gov.au/publications/investigation_reports/2...

( Final Report TAB )

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#82

Post by roncesvalles »

rawgabbit wrote: post #3

I wish to know what kind of system NOTAM runs on. An IBM 360 with tape backup?

Actually, it would've been pretty reliable if it ran on an IBM mainframe. That's their entire selling point.

There are two fundamental philosophies in fault tolerant systems. One is designing fault-tolerant hardware and running non-fault-tolerant software on it. This is what mainframes do. Practically any component of a mainframe can be hotswapped without shutting down the OS.

The other is designing fault-tolerant software and running it on non-fault-tolerant ("commodity") hardware. The latter is so popular that it's pretty much the default now, but it's not the only way of doing things.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#83

Post by kortilla »

dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

I see this a lot with people are experts in real time operating systems environments, particularly in aviation/space stuff (maybe because that’s where I worked for a while).

They have excellent intuition around making things redundant to single pieces of hardware failing but don’t really grok making stuff resilient to wider failures.

Anything involving transaction logs, rollbacks, and plain old backups take a backseat to live hardware-redundant environments. “It’s OK though because we follow the NASA software development process which has a rigorous set of validation steps that prevent bugs.”

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#85

Post by reidjs »

themitigating wrote: post #77
maxbond wrote: post #20

Earlier quoted context omitted.

I suppose nothing, I'll just be miserable while I'm doing it because I hate airports. A personal preference.

What is your preference based on?

Probably he’s been to… any airport. Maybe they’re bearable if you can get into a lounge

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#86

Post by brewdad »

ericpauley wrote: post #43
mgsouth wrote: post #10

Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…

This could be due to some third party service caching the NOTAMs. Even in the US Foreflight had all notams available but just couldn’t fetch news ones.

I have no idea how often NOTAMs typically get updated but would that explain why flights were able to operate for a few hours before ultimately a ground stop was called for?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#87

Post by rmason »

maxbond wrote: post #16
toddm wrote: post #9

On a positive note, you could have been on Amtrak: https://www.nytimes.com/2023/01/11/us/amtrak-delayed-va-fl.h...

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

Are you aware that an Amtrak passenger train was stuck in South Carolina for 39 hours this month? Some of the passengers felt they were being held hostage. At least at an airport you can walk outside and catch an Uber.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#88

Post by brewdad »

xwdv wrote: post #30
maxbond wrote: post #16

Earlier quoted context omitted.

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

If Amtraks were more luxurious, I might agree with you, but these days you might as well be riding Snowpiercer (and I don’t mean the good parts).

My seat on the Amtrak Cascades in September was more comfortable and had about 6 inches more legroom than any airplane I’ve ever flown on. First or Business class on an international flight will be way better than a roomette on Amtrak however.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#89

Post by brewdad »

happyopossum wrote: post #78
SV_BubbleTime wrote: post #72

Earlier quoted context omitted.

[flagged]

WTF are you putting on your eggs that anyone can smell from more than a foot away? Seriously- a soft boiled egg (hard boiled more likely given the travel context here) is almost completely odorless…

They do have an odor to them, especially since most people tend to boil them to death but I can think of a hundred things more offensive in a confined space.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#90

Post by dmix »

jrockway wrote: post #80
dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

What's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software. Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. N…

This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you keep them in sync the distinction comes down to hardware and lower-end systematic issues.

But most orgs can't realistically have two distinct software systems. How do you create proper isolation or failure mechanisms between them?

I'm guessing this sort of thing is what you mean by their experience with ETOPS.

Post reply on HN