Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

71–80 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#71

Post by bpye »

catminou wrote: post #45

I recall years ago in the mid 2000s a moderately sized US ISP that sold internet access in office buildings to mostly businesses had a rather interesting outage. They had switches (Extreme switches - known for their purple color aka Barney the dinosaur switches) in the basement or many buildings and backhauled to a colo to the rest of their network. They were doing some mass network upgrade during the early hours mai…

> The culprit? They downloaded the OS image using FTP and forgot to set binary mode. Various other network kit vendors I recall would do some level of validation (and anyone downloading should be checking the hash). RCA sent to customers was vague and just said it was a corrupted image / failed upgrades.

I've always been paranoid about this sort of thing when applying BIOS or firmware updates, despite those almost always having checksum validation, but I guess my caution is not unwarranted.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#73

Post by dmix »

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient.

https://news.ycombinator.com/item?id=34338373

> ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was intended to prevent outages because it allows for seamless switching between the two channels without impacting air traffic control should a problem occur in the active channel. The Agency believed this redundancy would make it unlikely that a problem in one channel could migrate to the other channel. In addition, to provide additional backup capabilities during ERAM’s implementation, FAA planned to temporarily maintain EBUS, its pre-existing backup system, before phasing it out completely beginning in 2015, to rely solely on ERAM’s dual channels.

> However, problems experienced during and since ERAM’s implementation have shown that the system remains susceptible to dual channel failures. As a result, FAA decided to maintain EBUS much longer than intended because air traffic controllers currently rely on EBUS for backup. However, with FAA’s ongoing and planned upgrades to ERAM, which will span the next 7 years, EBUS will soon become incompatible with the new hardware. As such, FAA plans to begin phasing out EBUS in April 2019, leaving ERAM without a backup system to supplement the system’s redundant dual channels.

A $2 billion dollar Lockheed Martin system where a memory overflow takes down BOTH the main system and the backup system. Surely that's not a signal of fragility.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#74

Post by thrdbndndn »

noobermin wrote: post #19

Is this the first time this system has failed so spectacularly? While the failure is notable, it does speak to the uptime of this system.

I agree.

Not exactly that we need to praise them or anything, but I think sarcastic tone in some replies ("if only they did something BASIC") aren't needed.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#76

Post by defrost »

shemnon42 wrote: post #75

Was the file in ASCII when it should have been in EBCDIC?

That's just silly.

Obviously it was due to details being entered using the now standard International foot when the software expected the now obsolete US survey foot [1] .. the discrepency across a transcontinental flight was large enough for a pilot to fall through.

[1] https://www.nist.gov/news-events/news/2023/01/new-years-eve-...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#77

Post by themitigating »

maxbond wrote: post #20
robswc wrote: post #18

Earlier quoted context omitted.

What is stopping you from doing that in an airport?

I suppose nothing, I'll just be miserable while I'm doing it because I hate airports. A personal preference.

What is your preference based on?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#78

Post by happyopossum »

SV_BubbleTime wrote: post #72
toddm wrote: post #9

On a positive note, you could have been on Amtrak: https://www.nytimes.com/2023/01/11/us/amtrak-delayed-va-fl.h...

[flagged]

WTF are you putting on your eggs that anyone can smell from more than a foot away?

Seriously- a soft boiled egg (hard boiled more likely given the travel context here) is almost completely odorless…

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#79

Post by shemnon42 »

GalenErso wrote: post #11
uejfiweun wrote: post #2

I wonder if the corruption could have been caused by a cosmic ray bit flip. It would be funny to think that one single tiny subatomic particle knocked down our entire flight system.

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.

Cosmic rays are more common than you think. Google's early infrastructure was impacted by a supernova (because their nodes were so cheap). But something like NOTAM can handle these single bit flips without a problem.

https://www.newyorker.com/magazine/2018/12/10/the-friendship...

File this under "early Google's infrastructure was a low grade cosmic ray detector."

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#80

Post by jrockway »

dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

What's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software.

Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. Nobody thought to be as paranoid about software as they are about planes flying over the ocean, but honestly, paranoia is good if you want reliability.

Post reply on HN