Linux's fsync() woes are getting some attention
rhaas.blogspot.com
Linux's fsync() woes are getting some attention
1–10 of 69 posts
Re: Linux's fsync() woes are getting some attention
#2[deleted]
Re: Linux's fsync() woes are getting some attention
#3I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things:
* write ordering ("write B must hit disk after write A")
* notification ("let me know when write A is on disk")
In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words, what you want should be nearly free, but instead you have to do a very expensive operation.
For an example for notification: suppose I have a temporary file with data that is being journaled into a data store. The operation I want to do is: 1. Apply changes to the store 2. Wait until all of those writes hit disk 3. Delete the temporary file I don't care if step 2 takes 5 minutes, nor do I want the kernel to schedule my writes in any particular way. If you implement step 2 as a fsync() (or fdatasync()) you're having a potentially huge impact on I/O throughput. I've seen these frequent fsync()s cause 50x performance drops!
Re: Linux's fsync() woes are getting some attention
#4Interesting couple of related articles / rants by Jeff Darcy:
Re: Linux's fsync() woes are getting some attention
#5I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…
Not a kernel dev but to my knowledge "write B must hit disk after write A" would actually be rather easy to implement: if the storage device supports it: insert a write barrier.
For the "let me know when write A is on disk" adding a callback to a write-barrier would do, not sure what the kernel queue looks like though.
Re: Linux's fsync() woes are getting some attention
#6I think this is a very important area of improvements for linux. While we call it "multitasking" there are a lot of situations where one might doubt it deserves that title.
I have been experimenting with very low cost computing setups that optimize for robustness and that led me to pretty slow disk I/O. While thats not a typical scenario for desktop computing, it can and should be possible with the limited but sane resources I ended up with. In practice however, certain loads freeze the whole system until a single usually non-urgent write finishes. Basically the whole throuput is used for a big write and then X (and others) freeze because they are waiting for the filesystem (probably just a stat and similar).
There are differences between applications. Some "behave" worse than others. Some even manage to choke themselves (ever seen GIMP take over an hour to write 4MB to an NFS RAID with 128kb/s throughput?).
I guess this is a hard problem, but I would wish for an OS to never stall on load. It is even better to slow down exponentially than to halt other tasks. Ideally the sytem would be smart and deprioritize long-running tasks so that small, presumably urgent, tasks are impacted as little as possible.
Re: Linux's fsync() woes are getting some attention
#7I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…
Not a kernel dev but to my knowledge "write B must hit disk after write A" would actually be rather easy to implement: if the storage device supports it: insert a write barrier. For the "let me know when write A is on disk" adding a callback to a write-barrier would do, not sure what the kernel queue looks like though.
In principle, yes. I believe most modern filesystems use (kernel-level) write barriers to ensure filesystem consistency, and those do depend on having hardware barriers as well.
The kernel is not set up to export this interface, however, and there doesn't appear to be any serious work being done to fix that.
EDIT: http://pl.atyp.us/2013-11-fixing-fsync.html has more details.
Re: Linux's fsync() woes are getting some attention
#8I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…
Is that not the purpose of [aio_fsync](http://pubs.opengroup.org/onlinepubs/009696899/functions/aio...)?
Disclaimer: I've never used it, I just filed it in the back of my mind for if I wanted to do basically what you just described.
Re: Linux's fsync() woes are getting some attention
#9I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…
I don't understand, but I am not very knowledgable about this any of this. My understanding is that fsync is an instruction to command the physical disk to flush its write buffer to the non-volatile (platters/flash) portion of storage, so it is safe to yank the cord from the wall.
In your scenario, it seems like you could have a situation where the temp file and the data are sitting on RAM inside the HD.
I honestly don't see how multiple processes that share a single physical disk can't get trashed by a processes that instructs the disk to constantly flush the buffer. It seems like asking for a network interface that doesn't slow down the network due other processes.
Re: Linux's fsync() woes are getting some attention
#10It always amazes me that after all these years, Linux still hasn't fixed this.
In my experience, any program that overloads I/O will make the system grind to a halt on Linux. Any notion of graceful degradation is gone and your system just thrashes for a while.
My theory about this has always been that any I/O related to page faults is starved, which means that every process spends its time slice just trying to swap in its program pages (and evicting other programs from the cache, ensuring that the thrashing will continue).
I've never gotten hard data to prove this, and part of me laments that SSDs are "fast enough" that this may never actually get fixed.
Can anyone who knows more about this comment? It seems like a good rule inside Linux would be never to evict pages that are mapped executable if you can help it.
Has anyone experimented with ionice or iotop? http://www.electricmonk.nl/log/2012/07/30/setting-io-priorit...