141–150 of 402 posts
Earlier quoted context omitted.
What kind of database are they using, I wonder, to end up with such a spectacular failure?
Well, for example, MySQL/MariaDB using utf8 tables will instantly go down if someone inserts a single multibyte emoji character, and the only way out is to recreate all tables as utf8mb4 and reimport all data.
In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not c…
In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not c…
Earlier quoted context omitted.
The technology to detect and recover from disk failures does exist. RAID and ZFS, for example. I would not expect a disk failure to replicate to the backup.
Working with critical infrastructure and lack of true in depth oversight it wouldn’t surprise me DR plans were not ever executed or exercised in a meaningful manner.
Comprehensive DR testing is really difficult. Many orgs settle for “on paper,” or “in theory” substitutions for real testing.
They do it right; no problem.
Doing it right, though … there’s the rub …
> In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system -- a significant decision, because the reboot can take about 90 minutes, according to the source. 90 minutes; this certainly appears to be an advertisement about Windows OS.
I was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?
I think a big takeaway from it is that designing systems which are failure-free is a fool's errant - no matter how hard you try, you can never get rid of 100% of the bugs.
Instead, make it failure-tolerant: sooner or later every part of the system will break, so it should be constructed in such a way that it can gracefully recover from failures, and even operate with some parts of it unavailable. Crashes are expected, so the system is designed to handle them properly.
No. NOTAM is "notice to airmen."
Quote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Can…
"...central database for all NOTAMs (Notice to Air Missions) nationwide" No. NOTAM is "notice to airmen."
"...central database for all NOTAMs (Notice to Air Missions) nationwide" No. NOTAM is "notice to airmen."
Nope. https://www.faa.gov/about/initiatives/notam/what_is_a_notam