131–140 of 402 posts
Earlier quoted context omitted.
What kind of database are they using, I wonder, to end up with such a spectacular failure?
Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.
AFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans. Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required? From 50k feet, it looks like something that c…
> cryptographically-authenticated massively-replicated virtual data structure that's oblivious about, and robust against all sorts of failure in, the exact systems, languages, update-paths, etc used to keep it in sync or implement any one user's view. All that pilots need to know is: "I have a full local copy, signed by the right update-authorities, as of roughly-now." You've solved one problem and created several mo…
Especially for a simple log-like system, with a limited number of permissioned authorities.
Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…
I wish to know what kind of system NOTAM runs on. An IBM 360 with tape backup?
Actually, it would've been pretty reliable if it ran on an IBM mainframe. That's their entire selling point. There are two fundamental philosophies in fault tolerant systems. One is designing fault-tolerant hardware and running non-fault-tolerant software on it. This is what mainframes do. Practically any component of a mainframe can be hotswapped without shutting down the OS. The other is designing fault-tolerant so…
How would an IBM mainframe help you with a corrupted database file? I understand that reliable hardware makes the corruption less likely to happen for hardware reasons, but it can also be the result of a software bug, or some unexpected and not correctly checked input.
Earlier quoted context omitted.
> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.
What kind of database are they using, I wonder, to end up with such a spectacular failure?
Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
What's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software. Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. N…
you are pointing out that they prioritized that planes in the air could land safely over allowing more planes to take off. I'm actually quite reassured now.
Earlier quoted context omitted.
This is highly informative. No seriously, it's probably the best thing I've read on NOTAMs. https://deathtonotams.com/ from there you can get to https://fixingnotams.org/ and pilots won't stop laughing and crying when they read https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...
And the answer to the question, "How many?" About 1.5 million per year.
So we're talking, (1.5M * 80 bytes=) 120MB a year, uncompressed. (With the tiny controlled-vocabulary, maybe 6x compression possible?)
Every pilot could have a RAM-resident local queryable copy in a their commodity handheld devices.
AFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans. Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required? From 50k feet, it looks like something that c…
This is highly informative. No seriously, it's probably the best thing I've read on NOTAMs. https://deathtonotams.com/ from there you can get to https://fixingnotams.org/ and pilots won't stop laughing and crying when they read https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...
Earlier quoted context omitted.
Why would this be an indictment of any specific database technology? If your disk fails and corrupts the filesystem, you're toast, regardless of what database you are using.
journal databases are specifically designed to avoid catastrophic corruption in the event of disk failure. the corrupt pages should be detected and reported by the database will function fine without them
That's different from filesystems that do checksumming (zfs, btrfs). Those can detect corruption.
In any case, if you use a database it handles these things by itself (see ACID). However I don't believe they can necessarily detect disk corruption in all cases (like checksumming file systems).
Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
I spent a lot of time at the FAA writing software. (6+ years) there is a huge culture of process, policy and not a whole lot of thinking or analysis or actually understanding the problems that they are working on. it is maddening. imagine a spreadsheet with 700 lines in it telling you that you need to do ABCDEFG each of those lines is instructing you to write a document detailing a procedure with the chain of custody…