Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux's fsync() woes are getting some attention

rhaas.blogspot.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Linux's fsync() woes are getting some attention

#31

Post by rosser »

dap wrote: post #21
jerf wrote: post #20

Earlier quoted context omitted.

Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now " guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. In fact filesystems in general attempt to implement this for themselv…

> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a pr…

You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on.

No, you can't. At least not without knowing what you're doing and careful planning.

My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather powerful IO subsystems — my production environment has SAS SLC SSDs under hardware RAID with a ginormous cache. I still see the behavior described in TFA far more often than I'd like. Linux really is pretty dumb here.

For example, because of this fsync() issue, and the fact that fsync() calls flush all outstanding writes for the entire filesystem the file(s) being fsync()'ed reside upon, I've set up my servers such that my $PGDATA/pg_xlog directory is a symlink from the volume mounted at (well, above) $PGDATA to a separate, much smaller filesystem. (That is: transaction logs, which must be fsync()'ed often to guarantee consistency, and enable crash recovery, reside on a smaller, dedicated filesystem, separate from the rest of my database's disk footprint.)

If I didn't do that, at every checkpoint, my performance would measurably fall. I learned this lesson the hard way, at an old job, where my postgres clusters lived on a SAN — it wasn't just my db instances that were being adversely affected by this IO storm. It was everything else that lived on the filer, too.

That's how bad it can be.

Re: Linux's fsync() woes are getting some attention

#32

Post by fiatmoney »

est wrote: post #17

Hmm. maybe that's why they had so many issues with single instance Redis & Mongodb? As soon as fsync() the whole db became unresponsive.

MongoDB's storage engine is more or less a mmap'd linked list of documents. It has a lot of issues once you actually start doing reads or writes, whether it's because your working set exceeds RAM or you actually want durability. It's a nice term-paper DB implementation but there's a good reason why most of the big single-instance RDBMS's use their own I/O algos instead of delegating to the kernel. Fundamentally the kernel doesn't know your optimal access pattern or your desired tradeoffs.

Redis turned away mmap'd storage some time ago; you can snapshot the DB to disk, but that's done via a fork() and the writes happen in that other process.

Re: Linux's fsync() woes are getting some attention

#34

Post by dap »

rosser wrote: post #31
dap wrote: post #21

Earlier quoted context omitted.

> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a pr…

You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. No, you can't. At least not without knowing what you're doing and careful planning. My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather…

You're absolutely right. The situation on Linux sounds bad, and it sounds like work's being done to improve that. Some other systems handle this case fine, though, and the problem isn't with the fsync() interface.

Aside: I worked on one of those filers, and I've seen some shit too. :) Firmware, especially disk firmware, was the worst.

Re: Linux's fsync() woes are getting some attention

#35

Post by sanxiyn »

bodyfour wrote: post #3

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…

Featherstitch http://lwn.net/Articles/354861/ is an interface (and an implementation) for write ordering without blocking. As far as I can tell, it went nowhere.

Re: Linux's fsync() woes are getting some attention

#36

Post by SamReidHughes2 »

dap wrote: post #29
SamReidHughes2 wrote: post #24

Earlier quoted context omitted.

> I don't know what the intended use case actually is, Generally speaking, if you've got a database system of some sort and want to write data to a file. > It's still not enough: with write ordering and notification but no instruction to actually write the data soon, the kernel can buffer it indefinitely. That's not really the problem -- being able to say "do this write operation after this other write operation" wou…

>> I don't know what the intended use case actually is, >Generally speaking, if you've got a database system of some sort and want to write data to a file. But database systems have been around for years without such an interface, and can't they basically saturate a storage subsystem?

You can always saturate a storage subsystem -- add more clients (assuming you don't saturate the CPU, the CPU's memory bandwidth, or the network interface -- any of which can happen if you put a high-end storage device on otherwise typical hardware). But what you get is higher than the minimal possible latency.

For example, suppose you send a bunch of write operations to the disk and then send an fsync. Well, if those write operations happen one after the other (figuratively) (because there's a bunch of them), their actual completion time would on average be half that of the actual waiting time all of them must suffer through.

Now suppose you've got the ability to do fine-grained fsyncs on particular write operations, efficiently. It would still be useful and result in improved latency if the disk or OS knew that getting block A on disk didn't matter to the process until block B was also on disk, and took advantage of that fact. And it would be extra-useful if the disk or OS knew that block B had to be written after block A, because then you would be able to save a round-trip or save on CPU bandwidth necessary for marking or checksumming blocks to a sufficient degree that you can determine upon startup whether they were completely and correctly written.

Re: Linux's fsync() woes are getting some attention

#37

Post by bodyfour »

dap wrote: post #12
bodyfour wrote: post #3

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…

> In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could li…

> you could literally be waiting forever

Technically that's true, but it's not an issue in real life. Any sensible operating system will flush dirty buffers to disk eventually. Imagine if a sudden power outage lost writes you had made a month ago; it would be considered a pretty severe OS error.

So in reality, you might be waiting many seconds for the write to hit disk, but you won't be waiting hours.

> That's exactly what fsync() is supposed to do.

No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. Oh, and block my thread until you're finished." It's an expensive operation.

Re: Linux's fsync() woes are getting some attention

#38

Post by tytso »

rosser wrote: post #31
dap wrote: post #21

Earlier quoted context omitted.

> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a pr…

You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on. No, you can't. At least not without knowing what you're doing and careful planning. My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather…

It's not true that fsync() calls flush all outstanding writes for the entire file system; that was true for ext3 in data=ordered mode, but it's definitely not true for ext4 or xfs. If you use fdatasync(), and there were no write commands that issued against the file descriptor that required metadata updates (i.e., you didn't do any block allocations, etc), then both ext4 and xfs won't need to trigger a journal commit, so the only thing that has to get sent to disk is all of the dirty metadata blocks, followed by a SYNC CACHE command which forces the disk drive to guarantee that all writes sent to the disk will survive a power cut.

If you use fsync() and/or you have allocated blocks or otherwise performanced a write which required updating file system metadata, and thus will require a journal commit, then you will need to force out all pending metadata updates to the journal as part of the file system commit, but that's still not the same as "flush all outstanding writes for the entire file system".

Re: Linux's fsync() woes are getting some attention

#39

Post by deathanatos »

Qantourisc wrote: post #5
bodyfour wrote: post #3

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words,…

Not a kernel dev but to my knowledge "write B must hit disk after write A" would actually be rather easy to implement: if the storage device supports it: insert a write barrier. For the "let me know when write A is on disk" adding a callback to a write-barrier would do, not sure what the kernel queue looks like though.

While a "notify me when this write completes" (but don't feel the need to rush) might be useful, I think it'd be more useful to be able to push both write A and write B to the kernel, along with the condition of "A before B". You not only get the write ordering, but the kernel gets full knowledge of what's going on. Perhaps it could use this to write A and B together (but journaled at the FS level) to guarantee the order, whereas a simple write-complete notification implies at least two writes, and the latency of doing such.

Re: Linux's fsync() woes are getting some attention

#40

Post by deathanatos »

dap wrote: post #21
jerf wrote: post #20

Earlier quoted context omitted.

Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now " guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. In fact filesystems in general attempt to implement this for themselv…

> Anything where you want an image to be guaranteed consistent, even if not complete, could use an ordering guarantee without a particular "this has been written now" guarantee. A log-structured data store where you don't mind a bit of data loss if there's a power outage is a particularly clear example of that, but it's a useful property in general. So a little bit of data loss is okay, but a lot isn't? How does a pr…

> So a little bit of data loss is okay, but a lot isn't?

I don't think that's the issue being put forth here, all the way to the most-ancestor comment. It's not between a bit of data loss and a lot of data loss: if the power fails, I fully expect that some data that was in buffer might not have made it to the disk. But that's okay: I left my file in a state that's still valid (using a journal or something), and I can recover from it.

But if write B happens before write A, my file is corrupt, and it's game over. I don't need to override the kernel's I/O scheduler to say "write this write right now!", I just need to ensure that A goes before B. That's it; it can happen whenever, but it must happen in that order.

Currently, the only solution seems to be fsync and forcing everything to disk right now, and screwing over any buffering what-so-ever¹. I just want to specify to the kernel, "here's my data, here's how it must be written, but you can buffer and write it when convenient, as you have a better picture of the whole system I/O."

¹It's funny that the article mentions Postgres/MySQL, as I've heard of fsync woes through browsers using SQLite.

Post reply on HN