Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

61–70 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#61

Post by drpixie »

gojomo wrote: post #41

AFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans. Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required? From 50k feet, it looks like something that c…

The abbreviations way pre-date SMS - they were standardised back in the teletype days, 1940s-1950s, when printing speed was 30-100cps! Now NOTAMs (mostly) comply with global standards so, like VHF AM aviation radio, substantial changes are impractical.

There is significant debate concerning the number (too many) and size (too long) of NOTAMs, but I can assure you that when you get an unexpected rerouting on a dark, bumpy, busy night, you do not want to be reading pages on plain-text prose! All experienced pilots are comfortable with the cryptic NOTAMs, and are used to scanning the abbreviations for important items :)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#62

Post by tanelpoder »

mdavidn wrote: post #51
lopkeny12ko wrote: post #49

Earlier quoted context omitted.

Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

The technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.

Yep and if you ship WAL transaction logs to standby databases/replicas, corrupt blocks or lost writes in the primary database won't be propagated to the standbys (unlike with OS filesystem or storage-level replication).

Edit: Should add "won't be silently propagated"

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#63

Post by ct520 »

mdavidn wrote: post #51
lopkeny12ko wrote: post #49

Earlier quoted context omitted.

Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

The technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.

Working with critical infrastructure and lack of true in depth oversight it wouldn’t surprise me DR plans were not ever executed or exercised in a meaningful manner.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#64

Post by Johnny555 »

ecuzzillo wrote: post #54
Johnny555 wrote: post #47

Earlier quoted context omitted.

Well, sometimes it's because you already boarded and started to taxi and there are so many planes on the ground that there are no gates open to get back to the terminal so you spend hours on the plane waiting for a slot to open so you can get off, meanwhile the galley runs out of food/drink and the toilets fill up. Then when you finally get into the terminal, it's chaos, no one knows when planes will be flying again…

Southwest?

It was United or Alaska and was a few years back and fortunately it wasn't me on the plane, it was my wife - a big east coast storm and airport closures diverted her flight (along with a bunch of other flights).

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#65

Post by kqr »

m463 wrote: post #34
eddsh1994 wrote: post #32

I find it quite weird multiple things failed on the same day, such as the UK post office system unable to deliver abroad

I suspect this is just statistics, like "it is likely that two people will have the same birthday in a roomful of people"

The other classical example being the large number of Prussian soldiers killed by horse kicks in the late 19th century.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#66

Post by tenken »

birdyrooster wrote: post #50
tigerBL00D wrote: post #35

Earlier quoted context omitted.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Imagine you have one node which is running as a replica of another and it takes the backups. Well, let’s pretend it is backing up the corrupted data once in a while and it happened to overwrite their cold backup. They could have any number of databases and still had this failure. It’s more their methodology for taking backups. They should have many points in time to choose from to rebuild their database. They should…

> They should be testing their databases before backing them up blindly.

Oh you mean they should be testing/validating the generated backup db file before replicating it to long-term archive ...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#68

Post by jdhendrickson »

catminou wrote: post #45

I recall years ago in the mid 2000s a moderately sized US ISP that sold internet access in office buildings to mostly businesses had a rather interesting outage. They had switches (Extreme switches - known for their purple color aka Barney the dinosaur switches) in the basement or many buildings and backhauled to a colo to the rest of their network. They were doing some mass network upgrade during the early hours mai…

I had a boss purchase those switches without consulting me, he liked the color. They were pure garbage. Thanks for reminding me of that, I'll have to ping a former co-worker and have a laugh.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#69

Post by marrone12 »

mgsouth wrote: post #10

Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…

The CBC is great. I watch the news show every night even as an American.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#70

Post by readthenotes1 »

tenken wrote: post #66
birdyrooster wrote: post #50

Earlier quoted context omitted.

Imagine you have one node which is running as a replica of another and it takes the backups. Well, let’s pretend it is backing up the corrupted data once in a while and it happened to overwrite their cold backup. They could have any number of databases and still had this failure. It’s more their methodology for taking backups. They should have many points in time to choose from to rebuild their database. They should…

> They should be testing their databases before backing them up blindly. Oh you mean they should be testing/validating the generated backup db file before replicating it to long-term archive ...

Way back when use cases were a thing, I used to chide people for saying that Backup was a use case.

No, Restore is a use case.

(Replace "use case" with "requirement" or "user story"...)

Post reply on HN