Linux's fsync() woes are getting some attention

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants.

If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could literally be waiting forever. If the function ever returned, it would only be because something else on the system asked for a sync (and it was easier to write everything), or because you're operating on a filesystem that chooses to write data sooner than required. Both of these are basically working by accident (i.e., may well not work on other POSIX systems).

I think what you really want is to instruct the kernel that this data should be written out, and you want to block until that happens. That's exactly what fsync() is supposed to do. The fact that some kernels hammer the I/O subsystem while doing so is a bug in those kernels, not the fsync() interface.

What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification.

fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written.

Happens to Windows and Mac OS too. Every time I boot Dropbox thrashes my disk for 10 minutes while the system is almost completely unresponsive.

> What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification.

It's still not enough: with write ordering and notification but no instruction to actually write the data soon, the kernel can buffer it indefinitely.

If you want the data on disk, use fsync(). If you don't care, don't. If the problem is that you can't afford the latency imposed by the multiple fsync() calls required to ensure correct data ordering for your application, fine. But that's not the problem the OP talks about. That was about fsync() hammering the I/O subsystem. You can solve that problem by fixing the fsync() implementation.

> fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written.

If you really want that, could you mmap(2) the file and use msync(2)?

For this as well as the other cases described (e.g., wanting write ordering), I don't know what the intended use case actually is, but is it possible that there's another way to organize the data that's still correct and performs well without changing the POSIX interface? It seems likely, given the number of different programs out there that manage to get by with it, and there's a rather significant cost to adding a new interface.

(One option is to write everything you need into a temporary file, fsync() it, then synchronously rename it to "commit" it. That still requires two fsync's, but never more than that. You can generalize this for multiple files using a temporary directory.)

FWIW, I typically work on illumos systems. On fsync(), ZFS records only an intent log record. That alone helps, since it's not stopping the world to write out everything that's been buffered. For particularly latency-sensitive applications, we use a separate intent log device on an SSD. (Regardless of write ordering and filesystem optimizations, an SSD is necessary in order to guarantee something is on stable storage with latency better than spindles can provide.) This configuration works very well.

Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general.

In fact filesystems in general attempt to implement this for themselves, because a filesystem should ideally always be in a consistent state... it may not be the right state, per se, but it should not be actively inconsistent, leaving you (or fsck) to basically guess what the correct state is.

The problem appears to be that today there's only "write this all out now and DO ABSOLUTELY NOTHING ELSE until that happens", and "yeah, whatever, write it whatever order you like and I sure hope it all works out."

Is that correct? There really isn't anything like a write barrier? All my reading of the links here seem to indicate that but I find it hard to believe that really is an accurate summarization of the current state of affairs on Linux. (Though I concede that I can see how hard it would be to propagate such a guarantee all the way from the disk hardware, through the drivers, through a large and varied number of file systems, all the way out to user space, without bugs, bugs, bugs.)