Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux's fsync() woes are getting some attention

rhaas.blogspot.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Linux's fsync() woes are getting some attention

#41

Post by bodyfour »

dap wrote: post #21
jerf wrote: post #20

Earlier quoted context omitted.

Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now " guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. In fact filesystems in general attempt to implement this for themselv…

> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a pr…

> So a little bit of data loss is okay, but a lot isn't?

No data loss is OK. However, delaying acknowledgement to the network client requesting the db update often is OK. Especially if your protocol allows many simultaneous requests to be in flight with asynchronous completion notification. (Similar to how TCQ on a hard drive works)

Right now it's very hard to do this type of workload efficiently. That's a pity because with write-notifications you could do it with zero extra I/O cost.

Re: Linux's fsync() woes are getting some attention

#42

Post by bodyfour »

tytso wrote: post #38
rosser wrote: post #31

Earlier quoted context omitted.

You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. No, you can't. At least not without knowing what you're doing and careful planning. My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather…

It's not true that fsync() calls flush all outstanding writes for the entire file system; that was true for ext3 in data=ordered mode, but it's definitely not true for ext4 or xfs. If you use fdatasync(), and there were no write commands that issued against the file descriptor that required metadata updates (i.e., you didn't do any block allocations, etc), then both ext4 and xfs won't need to trigger a journal commit…

> or you have allocated blocks or otherwise performanced a write which required updating file system metadata

What if you're appending to a file, and want to checkpoint every so often. I guess you can be clever with fallocate(FALLOC_FL_KEEP_SIZE) to avoid the block allocation, but won't st_size still need to be updated?

I also assume that st_mtime doesn't count towards dirtying the metadata.

Re: Linux's fsync() woes are getting some attention

#43

Post by SamReidHughes »

maxlybbert wrote: post #28
SamReidHughes2 wrote: post #14

Earlier quoted context omitted.

What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification. fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written.

> fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written. I believe fsync is per-file (or, really, per-file descriptor). I wasn't familiar with this specific problem, but I believe the problem is that for fsync to work, it has to issue commands to the hardware to flush hardware buffers to d…

Well even doing it on a file is bad.

It's per file, but also, it's (necessarily) the in-core state of that file, as a Linux man page says.

So if you do a write on one thread, wait for that write to complete, and then signal another thread to do an fsync, I don't really know what happens.

Re: Linux's fsync() woes are getting some attention

#44

Post by otterley »

bodyfour wrote: post #37
dap wrote: post #12

Earlier quoted context omitted.

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could li…

> you could literally be waiting forever Technically that's true, but it's not an issue in real life. Any sensible operating system will flush dirty buffers to disk eventually. Imagine if a sudden power outage lost writes you had made a month ago; it would be considered a pretty severe OS error. So in reality, you might be waiting many seconds for the write to hit disk, but you won't be waiting hours. > That's exactl…

> No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. ..."

POSIX 1003.1 (2004) doesn't go so far as to require unrelated data to be committed (emphasis mine):

"The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes." (http://pubs.opengroup.org/onlinepubs/009695299/functions/fsy...)

I think there's a good argument that Linux and other workalike OSes go too far in flushing all dirty pages to disk when fsync() is called.

sync(), on the other hand, does require all dirty pages to be committed, though, ironically, it is permitted to return asynchronously (http://pubs.opengroup.org/onlinepubs/009695299/functions/syn...).

Re: Linux's fsync() woes are getting some attention

#45

Post by jamesaguilar »

ryandrake wrote: post #13
dap wrote: post #12

Earlier quoted context omitted.

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could li…

Coming from a graphics background, this reminds me of a similar problem in graphics. In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all co…

There is already fflush, and it is generally not what people calling fsync want.

Re: Linux's fsync() woes are getting some attention

#46

Post by PhantomGremlin »

FigBug wrote: post #16
haberman wrote: post #10

It always amazes me that after all these years, Linux still hasn't fixed this. In my experience, any program that overloads I/O will make the system grind to a halt on Linux. Any notion of graceful degradation is gone and your system just thrashes for a while. My theory about this has always been that any I/O related to page faults is starved, which means that every process spends its time slice just trying to swap i…

Happens to Windows and Mac OS too. Every time I boot Dropbox thrashes my disk for 10 minutes while the system is almost completely unresponsive.

Yes, OS X isn't very advanced.

In its heyday, Solaris was outstanding in terms of being responsive while simultaneously doing large amounts of I/O. (Or at least that's my perhaps clouded recollection, I haven't used Solaris in over 5 years).

Re: Linux's fsync() woes are getting some attention

#47

Post by riffraff »

dap wrote: post #12
bodyfour wrote: post #3

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could li…

could you explain what the kernels that "do it right" do differently from those that "hammer the I/O subsystem"?

Not being an expert, it would seem to me there shouldn't be much difference but obviously there is (i.e. TFA also says "linux woes").

Re: Linux's fsync() woes are getting some attention

#48

Post by zurn »

Good to see this summit involving the kernel developers, since their past situation sounds rather bleak interaction-wise: using kernel version from 2009 and haven't tested the improvements in the (2012) 3.2 kernel.

BTW, Linux provides the direct I/O O_DIRECT interface that allows apps to bypass the kernel caching business altogether. This is also discussed in Mel Gorman's message that this blog borrows from.

Re: Linux's fsync() woes are getting some attention

#49

Post by zurn »

jamesaguilar wrote: post #45
ryandrake wrote: post #13

Earlier quoted context omitted.

Coming from a graphics background, this reminds me of a similar problem in graphics. In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all co…

There is already fflush, and it is generally not what people calling fsync want.

fflush is unrelated to the business of flushing data to disk. It's just a libc FILE* thing. It causes libc to empty the (typically tiny) stdio buffer associated with the FILE* using the write() system call.

Re: Linux's fsync() woes are getting some attention

#50

Post by bodyfour »

otterley wrote: post #44
bodyfour wrote: post #37

Earlier quoted context omitted.

> you could literally be waiting forever Technically that's true, but it's not an issue in real life. Any sensible operating system will flush dirty buffers to disk eventually. Imagine if a sudden power outage lost writes you had made a month ago; it would be considered a pretty severe OS error. So in reality, you might be waiting many seconds for the write to hit disk, but you won't be waiting hours. > That's exactl…

> No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. ..." POSIX 1003.1 (2004) doesn't go so far as to require unrelated data to be committed (emphasis mine): "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device…

True, and it's good of you to clarify. However, what if the "other pending writes" are to the same file descriptor? Just because I'm eager to know when one write happens doesn't mean I care about other writes I made elsewhere.

sync() is basically a relic. Historically it only existed so that a UNIX userland process could roughly control the rate that dirty buffers went to disk. This was the "update" daemon historically, known as "fsflush" on SYSV. At least on linux that's internal to the kernel these days.

Post reply on HN