Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

121–130 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#121

Post by rmrfchik »

dmix wrote: post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

Just my 2 cents: 2 billons are payed not for reliability but for size. Size of the system (and size of the team).

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#122

Post by KronisLV »

In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay".

So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not corrupted (e.g. display/playback fails). If you have compression as a part of the backup process, then the compressed archive should also be validated, that it's not corrupt and that the data can also be extracted.

Only then would checksums of the backup be actually useful, once you'd know that you don't just have a bunch of mostly useless binary data on hand. Sadly, the tooling just isn't there (e.g. CLI utilities for validating everything) and there are far more file formats than most people want to concern themselves with.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#123

Post by danuker »

f1refly wrote: post #118
pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

TDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.

How would you apply TDD to a backup system? By writing backup and restore tests? What if one test fails unreproducibly?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#124

Post by kilotaras »

noobermin wrote: post #19

Is this the first time this system has failed so spectacularly? While the failure is notable, it does speak to the uptime of this system.

It's a different tradeoff. Systems that fails and *executes recovery path* from time to time tend to be more resilient to catastrophic failures.

Ideally you would want the combination of both: system doesn't usually crash/fail, but you force it to do so regularly. See https://en.wikipedia.org/wiki/Chaos_engineering

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#125

Post by nix23 »

GalenErso wrote: post #11
uejfiweun wrote: post #2

I wonder if the corruption could have been caused by a cosmic ray bit flip. It would be funny to think that one single tiny subatomic particle knocked down our entire flight system.

Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.

>They're so rare,

They are not rare, just because you don't detect them (consumer hardware) means nothing.

>>study done by IBM in the 1990’s that referenced 1 cosmic ray bit flip per 256MB of memory per month

https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#126

Post by thombat »

maxbond wrote: post #110
nullify88 wrote: post #106

Earlier quoted context omitted.

> I will say the views are unbeatable It's hard to agree with that. There are some amazing train routes through Austria, Norway, Switzerland, Germany that keep me looking out of the window the whole way.

I love the views from the train too, but I feel like the view flying is something magical everyone should experience at least once in their life. The ground fading away, the tiny cars, glittering rivers, seeing it in reverse for landing. That is something I'll wax poetic about. But if I could take the train the rest of my life I would. Clearly the middle ground here is to bring back zeppelins. /s

No sarcasm needed for zeppelins: the idea of gliding over the landscape low enough to hear through the openable window what's happening below fast enough to be useful for a domestic voyage, and in reasonable comfort: What's not to like?

I was waiting for a Ryanair flight from Friedrichshafen when one of the Zeppelin NT craft flew past, heading over the Bodensee towards Switzerland. It looked simultaneously classic and futuristic, the sort of shiny utopia future of 1950s sci-fi book covers. I envied HARD.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#128

Post by efficax »

lopkeny12ko wrote: post #49
tigerBL00D wrote: post #35

Earlier quoted context omitted.

What kind of database are they using, I wonder, to end up with such a spectacular failure?

Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.

journal databases are specifically designed to avoid catastrophic corruption in the event of disk failure. the corrupt pages should be detected and reported by the database will function fine without them

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#129

Post by fock »

rmason wrote: post #87
maxbond wrote: post #16

Earlier quoted context omitted.

I will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.

Are you aware that an Amtrak passenger train was stuck in South Carolina for 39 hours this month? Some of the passengers felt they were being held hostage. At least at an airport you can walk outside and catch an Uber.

in some countries in Europe you actually can do that too from trains (and not only in Romania, where a family crossed the tracks (2 <4 y.o. kids, 4 pieces of luggage) to conveniently enter the parking lot, where their car was located)

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#130

Post by gabesullice »

pharmakom wrote: post #116

I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?

I think they are. When a crash happens, the cause is identified, we patch the bug, and redeploy the system. After that, the software cannot fail for the same reason again—it's less fragile than before.

A fragile thing is like a wine glass. Once it breaks, it cannot be restored to its original state and especially not made better than before.

However, if you're talking about patching the system while it's still running, check out "Stop Writing Dead Programs" from Strangeloop '22: https://youtu.be/8Ab3ArE8W3s

Post reply on HN