[deleted]
A corrupt file led to the FAA ground stoppage – also found in backup system
191–200 of 402 posts
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#192Earlier quoted context omitted.
What kind of database are they using, I wonder, to end up with such a spectacular failure?
Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.
Having dealt with utf8mb4 data being inserted into the utf8mb3 columns many many times in the past, I've never had a table "instantly go down". You either get silent truncation or a refusal to insert the data.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#193Earlier quoted context omitted.
Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.
I need more info about this.
In MySQL the `utf8` character set is originally an alias for `utf8mb3`. The alias is deprecated as of 8.0 and will eventually be switched to mean `utf8mb4` instead. The `utf8mb3` charset means it's UTF8 encoded data, but only supports up to 3 bytes per character, instead of the full 4 bytes needed.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#194Earlier quoted context omitted.
The technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.
Neither checks the checksum on every read as that would be performance-prohibitive. So "bad data on drive -> db does something with corrupted data and saves corrupted transformation back to disk" is very much possible, just extremely unlikely. But they said nothing about it being bad drive, just corrupted data file, which very well might be software bug or operator error
This is wrong, both ZFS and btrfs verify the checksum on every read.
It's not typically a performance concern because computing checksums is fast on modern hardware. Besides, historically IO was much slower than CPU.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#195It seems like every year or so we have some gigantic technology meltdown in this industry. Imagine something like this happening at the NYSE, CME, et. at. Or, simply think about the last time you heard about a nationwide credit/debit card outage... Why can't we have our national infrastructure systems running at least as reliably as the Amex network? These systems are all information clearinghouses at the end of the…
>> Or, simply think about the last time you heard about a nationwide credit/debit card outage...
Last year in Canada when Rogers had a meltdown.
E.g. https://dailyhive.com/vancouver/everything-impacted-rogers-o...
Bell goes down more rarely but also can take large swathes of nation out.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#196Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…
Agencies like the FAA are notoriously risk adverse. Basically the motivations of most employees seems like, "if I mess up once, I get fired; if I don't produce any movement, I can't get fired". Naturally, the output is glacial progress and the introduction of tons of safety-theatre procedures (that get the implementers promotions for "increasing a culture of safety").
We all know this from the subject the FAA regulates: flights. New unleaded gas gets forever to approve. Simple changes to instruments takes years of certification, leaving 1960s technology in place when clear improvements have happened in the last 60 years.
I always wondered what would happen if this culture were carried over to another space. We know what it looks like in medicine because the FDA has similar priorities. Rarely do we see it in tech, which is known to "move fast and break stuff". But here we get a glimpse of the dystopian crossover between FAA-procedure-culture and software engineering.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#197Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…
And you were a part of that culture ?
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#198Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
Note that this most recent outage (NOTAMs) had nothing to do with ERAM.
ERAM is employed the the 23 air route traffic control centers [1] throughout the nation as their primary operating system. If there were a system-wide outage of ERAM, the consequences would be magnitudes more consequential than any NOTAM outage. Basically every flight in the air and not close to a terminal facility would lose radar contact and controllers would be working blind, causing widespread chaos and likely many safety incidents. Non-radar air traffic control is a thing, but generally controllers do not have adequate training or currency to do it safely, and definitely not at anywhere near normal capacity.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#199In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not c…
Related: http://www.taobackup.com/
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#200Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works.
That’s not speaking poor of the engineers (which in my experience can be very good) but of the management and innovation culture of these agencies, which is too often terribly broken. They would say they are “risk averse” but as yesterday highlights their poor approach to this creates a ton of risk.