Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

41–50 of 161 posts

Re: Unix’s file durability problem

#41
post #18

I think the man page http://linux.die.net/man/2/fsync answers the questions the article asks about fsync(). fsync() = YES metadata, YES data, NO dir entry fdatasync() = NO metadata, YES data, NO dir entry fsync() of dir = NO metadata, NO data, YES dir entry

For Linux, but it's not portable across all Unixes. fsync is allowed to do nothing (1) and does not need to work on directories (2). On Mac OSX, fsync will not flush the disk cache, which can lead to data loss (3).

(1) http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs...

(2) http://austingroupbugs.net/view.php?id=672

(3) https://developer.apple.com/library/mac/documentation/Darwin...

Re: Unix’s file durability problem

#42
post #37
post #34

Earlier quoted context omitted.

If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store , and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstr…

Microsoft actually tried something similar with WinFS [0] in the 00's, but the project failed. [0] - https://en.wikipedia.org/wiki/WinFS

Oddly, Apple's Core Data framework achieves pretty much all the goals WinFS set out to achieve—but does so sitting (for no good reason, really) on top of a regular filesystem. It'd actually be pretty easy to flip things around: give Core Data a raw block device persistence backend, and then write a filesystem driver on top of it.

One of my recently-planned hobby projects was to write a FUSE filesystem for OSX that would take all the shell-level glop (LaunchServices UTI-associated verbs, NSFileManager attributes, Spotlight metadata, etc.) and stick it onto the files themselves as plain xattrs, so POSIX tools could effectively be made to work with shell-namespace "Documents" (including treating directory-packages as files, and traversing archive files and container-file-formats as if they were directory-packages.) It'd be quite interesting to combine the two experiments, actually; it'd result in something very much like a POSIX compatibility layer on top of a WinFS-like filesystem.

Re: Unix’s file durability problem

#43
post #34
post #22

Earlier quoted context omitted.

If you then expose that file system through a POSIX file system API, you have all of the same issues of underspecified or unclear behaviour that the article mentions.

If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store , and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstr…

How would it be different than the filesystem API?

There was a great lightning talk a few years ago that I can't seem to find where the author described an API for storing blobs in a hierarchical namespace. Of course, halfway through, it became clear that it's just the POSIX API: you can "open" handles to objects, "rename" them, remove them, and so on. You'd end a transaction with "fsync()". (Okay, that one's a little more complicated, but I don't think it's as hard as the OP claims, at least for single files. Multiple files are more complicated, but that problem is intrinsically more complicated.)

Re: Unix’s file durability problem

#44

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

> A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

So if I need to create a file with a timestamp, you are suggesting creating a SQLite database, a table, and inserting my timestamp there?

It's nonsensical to "fix" a Unix low-level syscall issue by using SQLite.

Re: Unix’s file durability problem

#45
post #28

Earlier quoted context omitted.

The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision that it is necessary to write to disk. How do you actually do that for sure? It's not write() or even necessarily fsync(). The author is finding this frustrating because there are several things to do that can seem arbitrary, random, and counterintuitive. His…

> How do you actually do that for sure? It's not write() or even necessarily fsync(). Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data. I had to do it a few times. Once it was a realtime-is…

That's still missing the point. Say you're writing a program that stores a log of transactions. People are trading resources and you log those transactions so you can find out who owns which resources. User A just gave 100 units to user B; you need to store this information and tell both users that the transaction is complete. How do you do that? Your program will do something like "write(t_log_fd, t_entry, t_bytes);". Can you now tell the users that the transaction is complete?

Re: Unix’s file durability problem

#46
post #43
post #34

Earlier quoted context omitted.

If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store , and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstr…

How would it be different than the filesystem API? There was a great lightning talk a few years ago that I can't seem to find where the author described an API for storing blobs in a hierarchical namespace. Of course, halfway through, it became clear that it's just the POSIX API: you can "open" handles to objects, "rename" them, remove them, and so on. You'd end a transaction with "fsync()". (Okay, that one's a littl…

Two key differences:

1. "MVCC": the ability to effectively get a handle on a static copy of the entire filesystem, perform mutations to many objects, and then submit the transaction, at which point it will either commit or rollback depending on whether any of those objects have been modified.

Windows actually has this: https://en.wikipedia.org/wiki/Transactional_NTFS allows for exactly the sort of MVCC I'm talking about—and presents a very different API than the POSIX filesystem one.

2. "Object store": as in, you don't interact with the filesystem by getting writable handles to the file's underlying backing store, where processes can effectively treat a file as persistent shared memory. Instead, you ask for a floating unnamed "buffer" object, fill it up, and then submit it to the filesystem to be atomically stored; or, vice-versa, you ask to retrieve a file, and are passed a buffer handle of the representation of the file at the point in time you asked for it (presumably, for optimization's sake, backed by copy-on-write mmap'ed pages from the latest MVCC-linearized copy of the file.)

We actually have this today as well, but in an implicit and half-assed way. Files below some size threshold, in modern filesystems, don't use any FS extents for backing, but are instead stored as part of their filesystem directory-entry node. This basically makes the filesystem into an object store for those files—but without actually exposing any guarantee to userland that those files will be read/written as atomic operations.

---

To be clearer, there's a problem with this fs-atop-object-store design, as I've stated it so far:

An object store doesn't support every use-case a filesystem does, and a filesystem implemented only in terms of an object-store wouldn't be good for some things because of that. It would be terribly expensive to emulate a block device on top of an object store, for example—you'd have to make each emulated block an object, and mutate blocks by transactionally adding an updated block and removing the old block. This means that it wouldn't make sense to keep mountable read-write disk images on such a filesystem; nor would it make sense to keep database backing stores there.

But both of those things are effectively things that take no advantage of existing on top of a filesystem in the first place—they do their own index-building, their own sparse-allocation, their own journaling, etc. Making the filesystem an object store just makes it obvious that for those use-cases, the filesystem itself is pure overhead.

So, along with the MVCC object-store, the other part of my hypothetical system is a "buffer store": basically like a logical-volume manager, an API that consumes physical block devices and exposes handles to (cheap) logical resizable block devices. The object store could be implemented in terms of one of those logical block devices, but otherwise would completely ignore the existence of it. The filesystem compatibility layer, on the other hand, would allow you to request that a given filesystem object be backed by a persistent buffer (newly-created logical block device) instead of an object; and then whenever you requested that object from the filesystem, you'd get an IO handle to the logical block device, instead of an IO handle to a transactionally-resubmittable copy-on-write mmap of some object.

Re: Unix’s file durability problem

#47

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

Do you know if they address I/O reordering within the scheduler? For example transaction implementations often require that writes (distinct file system calls) hit the disk in a particular order to guarantee a sane state for the database. Prime example is the GNU bug for gzip: http://bugs.gnu.org/22768 The the writes to the `foo.gz` file have to hit the disk before the unlink but the I/O scheduler can reorder these p…

SQLite doesn't care about I/O re-ordering, but does care that the various fsync style calls work. SQLite uses a separate journal so it can rollback/forward changes.

Re: Unix’s file durability problem

#48

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

SQLite just (two month ago) got a new synchronous level EXTRA which fsyncs the directory after deleting the journal. The default is FULL (https://www.sqlite.org/pragma.html#pragma_synchronous ). So you still have to think about those problems.

Re: Unix’s file durability problem

#50
Not sure if I understand the problem. You don't want in most cases to flush out file changes or dirty memory to disk, so you an batch those operations or write only final change and not all intermediary ones. You don't want to wait forever since RAM is volatile. This is why databases have setting for checkpoint intervals, you can set the % or time for dirty pages to be written, I thought the default was 30 seconds, doesn't the same idea apply for file buffers?
Post reply on HN