Unix’s file durability problem

The best solution here it to replicate data across multiple machines, and take good backups that you can restore from, and plan on corruption to happen.

You also need to assess exactly how much you really need those transactions. If they're financial transactions where each one could be millions of dollars of stock, then you probably need to care about this a bit. For most web transactions I think the risk/cost analysis here is that you don't need to worry about being perfect. You might lose the last one or two transactions in the case of a hard crash, but your customer service team should be able to handle that and fix it for the customer out of band. You really should consider if you have a business justifcation for worrying about perfection.

On the other hand, I did live through the ext2 era before ext3 was production ready, and fully async super-duper fast filesystems really are bad. They would regularly corrupt the disk and require rebuilds. That wasn't a data availability problem, but one or two of those a week was problematic when it came to the operational load (out of 400-800 servers that we managed at the time). We later scaled out ext3 to 30,000 servers with some sets of servers having 2,500 hosts of basically the same type of webserver, and while kernel crashes were a daily issue, corruption and rebuilding was relatively low. If you're not at Google/Amazon/etc and dealing with servers counts an order of magnitude higher than this, you don't really need to worry about it. ext3 or ext4 and fdatasync should be fine, and then apply proper levels of engineering principles to ensure that you don't stay offline for too long or lose too much data.

You are dealing with free commodity hardware and software that isn't ever going to be perfect. If you really needed to never lose a transaction you'd probably be buying some kind of awfully expensive mainframe system.

Oh and I do recall one case of filesystem corruption leading to a service being down for over a week and probably the loss of a multi-million dollar business deal. But in that case the software ran on a single box. The dev team that was responsible for it never saw that as a problem even though the ops/sysadmin teams kinda yelled at them about it. Then one day the RAID array lost its mind and the server was unrecoverable. When we attempted to rebuild it, it was discovered that over the years the software devs had tweaked the versions of libraries that their software linked against and by crashing all that information had been lost, so it took ages to debug and find the right incantations to get it all back up again. Huge business risk there, but nothing that could be mitigated by naval gazing analysis of filesystems and fdatasync -- backups, documentation, replication, proper config management practices, etc were what was needed.

Additionally, in Linux, you can get asynchronous fdatasync(2) with sync_file_range(2).

aio(7) is weird. It really only works with direct I/O (which is necessary much less often than people think it is), and IIRC aio_fsync(2) isn't implemented on Linux, but that doesn't matter so much because generally direct I/O implies synchronous I/O (but not always). See http://lse.sourceforge.net/io/aio.html

O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly.

How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.

"fsync" synchs the whole file, not just one I/O, which can take a while. Databases such as MySQL's InnoDB, which puts multiple tables in one file, can have independent I/O going on in different parts of a file.

aio(7) has the right mechanism, a callback/signal on completion. But it's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.

The original article complains that UNIX/Linux file system semantics aren't well enough defined for database safety. He's right. They're close, but not quite there, because behavior after a crash is unspecified.