Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

71–80 of 161 posts

Re: Unix’s file durability problem

#71

Post by jstimpfle »

skybrian wrote: post #64
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Your response pretty aptly demonstrates the problem in the original article. If you write to a temp file and then use rename, you need to pick a suitable temporary name (without a race condition) that other programs will ignore, and now you've probably created a garbage file if the program crashes before you get to the rename. You also need to make sure you've preserved the permissions in the original file, and hope…

As I said, if it's too unbearable a hack for you to unlink() the file directly after creat()ing it (to get an anonymous file handle that nobody else can see), for the unlikely case that the system crashes just in between, then just go ahead and use O_TMPFILE. It was made for you!

If you don't want to use O_TMPFILE (not super-portable) then just open(path, O_CREAT|O_EXCL). That fails should there be another file lying around, and you can try again with another random name.

Btw. you can also just mmap(2) some anonymous non-disk-backed memory, and possibly open_memstream(3) on it to get a FILE * if that suits you better.

Re: Unix’s file durability problem

#72

Post by zzzcpan »

> One issue is that unlike many other Unix API issues, it's impossible to test to see if you got it all correct and complete. If your steps are incomplete, you don't get any errors; your data is just silently sometimes at risk.

There is no problem with the API. The issue is in an underlying assumption, that data sometimes is not at risk. Sadly or maybe luckily, there is nothing you can do to guaranty 100% durability, no point trying to do the impossible. Your data is always silently at risk. Accept it, deal with it, minimize the risk, if you need to. Like replicate your data synchronously cross continent.

Re: Unix’s file durability problem

#73

Post by protomyth »

derefr wrote: post #34
jclulow wrote: post #22

Earlier quoted context omitted.

If you then expose that file system through a POSIX file system API, you have all of the same issues of underspecified or unclear behaviour that the article mentions.

If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store , and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstr…

You might look at probably the most popular system to ship with an object store, the Apple Newton. I would probably start there and work my up API wise.

Re: Unix’s file durability problem

#74

Post by Dylan16807 »

twa927 wrote: post #44
rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

> A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen(). So if I need to create a file with a timestamp, you are suggesting creating a SQLite database, a table, and inserting my timestamp there? It's nonsensical to "fix" a Unix low-level syscall issue by using SQLite.

"competes with" does not mean that it's better in all use cases.

But even in an extreme example like storing a single timestamp, there are situations where you want various guarantees that SQLite makes easy.

Re: Unix’s file durability problem

#75

Post by colanderman »

Animats wrote: post #67
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

sync_file_range(2) allows you to specify ranges to sync, instead of the whole file, but it's not that useful anyway.

In most DBs, you have separate data and log. The log is always only appended to, and the nature of disks is such that grouping concurrent transactions together into a single commit is a big win regardless of how syncing works. So syncing all outstanding I/O to the log file is generally what you want to do anyway.

While the data store is always being written to, its writes do not need to be synced to disk until a log checkpoint is made (and the old log subsequently discarded). And since that is an intermittent asynchronous process, there is not much to gain from finer-grained syncing here either.

See e.g. http://www.postgresql.org/message-id/4A51CB76.5020407@enterp...

Re: Unix’s file durability problem

#76

Post by colanderman »

jstimpfle wrote: post #71
skybrian wrote: post #64

Earlier quoted context omitted.

Your response pretty aptly demonstrates the problem in the original article. If you write to a temp file and then use rename, you need to pick a suitable temporary name (without a race condition) that other programs will ignore, and now you've probably created a garbage file if the program crashes before you get to the rename. You also need to make sure you've preserved the permissions in the original file, and hope…

As I said, if it's too unbearable a hack for you to unlink() the file directly after creat()ing it (to get an anonymous file handle that nobody else can see), for the unlikely case that the system crashes just in between, then just go ahead and use O_TMPFILE. It was made for you! If you don't want to use O_TMPFILE (not super-portable) then just open(path, O_CREAT|O_EXCL). That fails should there be another file lying…

Even easier than open(2) with a random file name is mkstemp(3).

Re: Unix’s file durability problem

#78

Post by rsync »

em3rgent0rdr wrote: post #16

I've been told to type sync into terminal whenever I want writes to complete.

"I've been told to type sync into terminal whenever I want writes to complete."

If you'd really like a guarantee, you can always unmount the filesystem, or alternatively, mount it read-only:

mount -ur /mnt/blah

That will guarantee, at least at the filesystem/OS level, that the writes are committed.

Re: Unix’s file durability problem

#79

Post by kentonv »

Animats wrote: post #67
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write.

Note that Linux ext4 does not do this. On a power outage, you can get bogus trailing zeros on a file which you were appending, because the file size was updated before the data was written. I asked Ted T'so about this and he said it was working as intended.

https://plus.google.com/+KentonVarda/posts/JDwHfAiLGNQ

Post reply on HN