Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

41–50 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#41

Post by gojomo »

AFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans.

Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required?

From 50k feet, it looks like something that could be replaced with a cryptographically-authenticated massively-replicated virtual data structure that's oblivious about, and robust against all sorts of failure in, the exact systems, languages, update-paths, etc used to keep it in sync or implement any one user's view.

All that pilots need to know is: "I have a full local copy, signed by the right update-authorities, as of roughly-now."

From a glance, NOTAM's uptime this year looks worse than Ethereum, but a bit better than Solana or Binance Smart Chain.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#42

Post by walrus01 »

gojomo wrote: post #41

AFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans. Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required? From 50k feet, it looks like something that c…

cryptic abbreviations in the NOTAM system reminds me of telcos using CLLI codes for unique geographical locations of physical telecom infrastructure sites - a legacy of 1960s mainframe stuff where the number of characters in a database field for text entry was extremely constrained.

https://en.wikipedia.org/wiki/CLLI_code

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#43

Post by ericpauley »

mgsouth wrote: post #10

Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…

This could be due to some third party service caching the NOTAMs. Even in the US Foreflight had all notams available but just couldn’t fetch news ones.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#44

Post by kgeist »

There was a similar situation in Russia just a year ago (a federal aviation agency not having access to its online system): https://mentourpilot.com/breaking-russian-aviation-authority...

>Due to temporary lack of access to Internet and malfunction of the electronic document flow system of Rosaviatsia the Federal Agency for Air Transport is switching to paper version.

>“The document flow procedure is being determined by the current records management instructions.

>“Information exchange will be carried out via AFTN channel (for urgent short message) and postal mail.

>“Please make this information available to all Civil Aviation Organizations.”

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#45

Post by catminou »

I recall years ago in the mid 2000s a moderately sized US ISP that sold internet access in office buildings to mostly businesses had a rather interesting outage. They had switches (Extreme switches - known for their purple color aka Barney the dinosaur switches) in the basement or many buildings and backhauled to a colo to the rest of their network.

They were doing some mass network upgrade during the early hours maintenance window and devices weren't coming back. I don't think they had OOB access to their boxes and it requires someone going out and recovering them after hours of downtime and getting techs there.

The culprit? They downloaded the OS image using FTP and forgot to set binary mode. Various other network kit vendors I recall would do some level of validation (and anyone downloading should be checking the hash). RCA sent to customers was vague and just said it was a corrupted image / failed upgrades.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#46

Post by ananonymoususer »

rubicks wrote: post #17
daniel-s wrote: post #5

Now if their ATC software ran on Emacs lisp they wouldn't have had any problem.

I understood that reference.

I didn't, so I looked it up. Interesting: https://news.ycombinator.com/item?id=33524683

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#47

Post by Johnny555 »

robswc wrote: post #18
maxbond wrote: post #16

Earlier quoted context omitted.

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

What is stopping you from doing that in an airport?

Well, sometimes it's because you already boarded and started to taxi and there are so many planes on the ground that there are no gates open to get back to the terminal so you spend hours on the plane waiting for a slot to open so you can get off, meanwhile the galley runs out of food/drink and the toilets fill up. Then when you finally get into the terminal, it's chaos, no one knows when planes will be flying again so you're not sure if you should stay there or try to get a hotel (which fill up quickly from all of the stranded travelers)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#48

Post by jquery »

gojomo wrote: post #41

AFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans. Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required? From 50k feet, it looks like something that c…

> cryptographically-authenticated massively-replicated virtual data structure that's oblivious about, and robust against all sorts of failure in, the exact systems, languages, update-paths, etc used to keep it in sync or implement any one user's view. All that pilots need to know is: "I have a full local copy, signed by the right update-authorities, as of roughly-now."

You've solved one problem and created several more problems.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#49

Post by lopkeny12ko »

tigerBL00D wrote: post #35
asveikau wrote: post #33

Earlier quoted context omitted.

> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#50

Post by birdyrooster »

tigerBL00D wrote: post #35
asveikau wrote: post #33

Earlier quoted context omitted.

> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Imagine you have one node which is running as a replica of another and it takes the backups. Well, let’s pretend it is backing up the corrupted data once in a while and it happened to overwrite their cold backup. They could have any number of databases and still had this failure. It’s more their methodology for taking backups. They should have many points in time to choose from to rebuild their database. They should be testing their databases before backing them up blindly.
Post reply on HN