Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

181–190 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#181

Post by ilyt »

mdavidn wrote: post #51
lopkeny12ko wrote: post #49

Earlier quoted context omitted.

Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

The technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.

Neither checks the checksum on every read as that would be performance-prohibitive. So "bad data on drive -> db does something with corrupted data and saves corrupted transformation back to disk" is very much possible, just extremely unlikely.

But they said nothing about it being bad drive, just corrupted data file, which very well might be software bug or operator error

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#182

Post by ilyt »

NavinF wrote: post #131
eurasiantiger wrote: post #98

Earlier quoted context omitted.

Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.

It doesn't block inserts with invalid data? I thought that was the whole point of telling the database what types you're using

It does and poster above is incompetent

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#183

Post by ilyt »

marginalia_nu wrote: post #143
KronisLV wrote: post #122

In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not c…

Seems like it would be prohibitively time-consuming. For a moderately large data-set (of say a few hundred Tb), by the time the backup of your data is vetted and validated, it's months old and effectively useless.

Yet you still need to do it, even if it is every few months, else you will eventually hit the snag.

If you don't have an option to restore the data your backup might as well not exist.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#184

Post by ilyt »

noobermin wrote: post #19

Is this the first time this system has failed so spectacularly? While the failure is notable, it does speak to the uptime of this system.

Uptime is kinda moot measure.

If it had uptime of 2 days but only went down for a minute at a time that would not be a problem

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#185

Post by jve »

From the article and comments I don't understand - are they talking about backup (copy of data) or backup system (nuke your primary system, failover to backup system and just keep working) ?

I think it is a SYSTEM

> It has a backup, which officials switched to when problems with the main system emerged, according to the source.

> Officials ultimately found a corrupt file in the main NOTAM system, the source told CNN. A corrupt file was also found in the backup system.

Perhaps the backup system is just using data from main system which currently is compatible but won't be in the near future?

Still there should be data copy somewhere, right (with corrupt data...)?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#186

Post by ilyt »

f1refly wrote: post #118
pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

TDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.

TDD not really. Having tests help, writing them first is irrelevant

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#187

Post by ilyt »

danuker wrote: post #123
f1refly wrote: post #118

Earlier quoted context omitted.

TDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.

How would you apply TDD to a backup system? By writing backup and restore tests? What if one test fails unreproducibly?

Well, you should test your backups. But TDD is entirely unrelated to the problem aside from including "test" in the name

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#188

Post by ilyt »

numbsafari wrote: post #12
uejfiweun wrote: post #2

I wonder if the corruption could have been caused by a cosmic ray bit flip. It would be funny to think that one single tiny subatomic particle knocked down our entire flight system.

My guess is some input system was updated and now allows Unicode, but the backend systems do not. Somebody inadvertently put a poop emoji in a text field.

Client not being able to put emojis finally found a budget for one legacy project to get some DB version upgrades...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#189

Post by atchoo »

nottorp wrote: post #170
pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

I haven't seen Taleb offer solutions... in the one book i managed to go through he just told you endlessly to be prepared for unpredictable events.

A long time since I read it but I think he gave examples like mithridatism, poisoning yourself to build up resistance. Anti-fragility is about incorporating change as an expectation instead of seeking stability and fearing it.

For software, that would mean practices like Chaos Monkey. If your production system stays up while an external process is constantly killing processes and deliberately corrupting memory and files then you have good confidence of riding through unexpected failures.

Post reply on HN