Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

121–130 of 161 posts

Re: Unix’s file durability problem

#121

Post by Animats »

Analemma_ wrote: post #119
asveikau wrote: post #109

Earlier quoted context omitted.

> NTFS doesn't do append correctly. Care to elaborate on that? I haven't seen issues with the two mechanisms I know of in NT: 1. CreateFile with FILE_APPEND_DATA 2. WriteFile with the OVERLAPPED offset set to ~0 Both of these get you writes that are as far as I know safe to have multiple processes writing to a common log file. Not sure about flushes to disk but you could probably get away with FlushFileBuffers for th…

NTFS handles append just fine. As another comment speculated, the parent may have meant NFS, which does notoriously have lots of problems with append, and many other POSIX semantics.

Sorry, meant NFS.

Re: Unix’s file durability problem

#122

Post by Animats »

skybrian wrote: post #64
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Your response pretty aptly demonstrates the problem in the original article. If you write to a temp file and then use rename, you need to pick a suitable temporary name (without a race condition) that other programs will ignore, and now you've probably created a garbage file if the program crashes before you get to the rename. You also need to make sure you've preserved the permissions in the original file, and hope…

Exactly. It takes a whole collection of workarounds to get well behaved file semantics across crashes, and the workarounds are different for Linux, BSD, and various other UNIX variants, and may differ depending on the file system type and system version.

Re: Unix’s file durability problem

#123

Post by marvy »

marvy wrote: post #100

I ran across a nice paper a few years ago: Rethink the Sync. https://www.usenix.org/legacy/event/osdi06/tech/nightingale/... The basic idea is that the file system provides two guarantees, one boring, and one interesting. I'll illustrate using code instead of words. The boring guarantee is: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document fclose(f); The system will try to make…

Looks like I remembered, but there aren't any comments.

Re: Unix’s file durability problem

#124

Post by beagle3 »

kentonv wrote: post #79
Animats wrote: post #67

Earlier quoted context omitted.

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. Note that Linux ext4 does not do this. On a power outage, you can get bogus trailing zeros on a file which you were appending, because the file size was updated before the data was written. I asked Ted T'so about this and he said it was working as intended. https://plus.google.com/+K…

It does happen with the default setting, but if you journal everything, then it doesn't happen. Performance on spinning rust, however, drops 50%.

Re: Unix’s file durability problem

#125

Post by dspeyer »

jstimpfle wrote: post #59
Animats wrote: post #54

I've suggested an approach to this before. There should be several types of files. - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. This should be the default when a file is opened via "creat()" - "Temp" files a…

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Rename doesn't always do what we want. The one serious investigation I've seen [1] unhelpfully states that "[W]hen a file is appended, and then renamed ... many file systems recognize it and allocate blocks immediately." Suggesting that some file systems don't, but failing to name them. In general, appends followed by directory operations are not order-preserved.

[1] https://www.usenix.org/system/files/conference/osdi14/osdi14...

Re: Unix’s file durability problem

#126

Post by planckscnst »

marvy wrote: post #100

I ran across a nice paper a few years ago: Rethink the Sync. https://www.usenix.org/legacy/event/osdi06/tech/nightingale/... The basic idea is that the file system provides two guarantees, one boring, and one interesting. I'll illustrate using code instead of words. The boring guarantee is: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document fclose(f); The system will try to make…

I misread this comment and it could be dangerous if someone who doesn't know misreads it the same way. So I want to put a big warning here.

Warning! This comment is _not_ saying that your operating system provides these guarantees. In fact, it almost certainly does not. This is a novel suggestion (and implementation?) presented in this particular paper.

Re: Unix’s file durability problem

#127

Post by planckscnst »

GauntletWizard wrote: post #104
schmichael wrote: post #82

Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware bugs and dead batteries. Those are just ways your storage hardware can fail you even if the kernel and libraries are bug free and you follow the right mystic incantations to sync data. While there's no excuse for bad documentation or poor APIs, you can never…

Yeah, from a SRE perspective, the last N writes are always purely probabilistic. The real quest is to have enough redundancy that that curve is fairly close to 1, and enough failure warning that your system will fix itself before it droops. That means Battery Backup, SMART detection, ECC memory, etc.

It's even worse than that. All writes have some probability of being incorrect, even after having been written to disk properly (the programmer did the right thing, the OS did the right thing, the filesystem did the right thing, and the disk did the right thing). After writing, your data can be modified by actions on other cells or by simply leaving it alone completely (see read disturbance and charge leakage). Some of this can be fixed by error correcting codes, but there is still a chance of loosing data simply by doing nothing wrong.

So yes, agreed++. You need to have a level of redundancy appropriate to the criticality of the data.

Re: Unix’s file durability problem

#128

Post by haberman »

avar wrote: post #120
haberman wrote: post #117

Earlier quoted context omitted.

> You have 100 concurrent web serving processes all writing to an access.log That makes no sense. Have each process write to its own log. This is making your life hard for no reason. > If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files This isn't about access.log, it's about transaction logs for database systems. Like the commit log for Po…

> That makes no sense. Have each process write to its own log. This is > making your life hard for no reason. This is literally how pretty much every daemon process works on Unix systems. You don't log in and tail /var/log/apache/access.log.{1..N} where N is the number of threads/processes. You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. Having to tail N logs where…

> You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics.

I can see how this would be convenient. That's fine and I have nothing against it. It's just not a problem I consider interesting, because there is no inherent need for all 100 writers to write to the same file. You could just as easily have each write to a separate file and have a separate process that merges them together if you want a single file for convenience.

> I really don't see how any of this is relevant to the original point of having "log" files.

Maybe you're not aware: every major database keeps a commit log, and writes it to a file called a log file (sometimes "write-ahead log file"). That is what I think of when I hear about "log" files in the context of file durability, because this is the case where file consistency and durability actually affect the consistency of the system. Here are some examples:

https://www.sqlite.org/tempfiles.html#walfile

http://www.postgresql.org/docs/9.1/static/wal-intro.html

https://leveldb.googlecode.com/svn/trunk/doc/impl.html

This is a much more interesting problem (to me) because it is harder and relevant to the consistency of any database system.

> Now you're talking about DB transaction logs and blockchains, which really don't need to have their nuances implemented in terms of POSIX file semantics. They can just have a single process that manages those expectations.

You do realize that SQLite can't just corrupt your database because you ran two SQLites concurrently, right?

SQLite has chosen to support concurrent SQLite processes all writing to the same database. Maybe you think they shouldn't do that, but they find it useful. And even in your world where they shouldn't do that, it's still not ok to just corrupt the database because the user didn't follow the rules.

> Everyone's expectations don't have to be implemented on the filesystem level

One of the kernel's key responsibilities is to arbitrate concurrent access to shared resources. It provides primitives that makes it possible to build higher-level abstractions. What I am describing is a primitive that allows for lock-free appends to consistent transaction logs. It could be useful for a lot of things, and makes at least as much sense as plenty of other things that are already in syscall interfaces.

Re: Unix’s file durability problem

#129

Post by marvy »

planckscnst wrote: post #126
marvy wrote: post #100

I ran across a nice paper a few years ago: Rethink the Sync. https://www.usenix.org/legacy/event/osdi06/tech/nightingale/... The basic idea is that the file system provides two guarantees, one boring, and one interesting. I'll illustrate using code instead of words. The boring guarantee is: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document fclose(f); The system will try to make…

I misread this comment and it could be dangerous if someone who doesn't know misreads it the same way. So I want to put a big warning here. Warning! This comment is _not_ saying that your operating system provides these guarantees. In fact, it almost certainly does not. This is a novel suggestion (and implementation?) presented in this particular paper.

Yes, definitely! If the edit timer hadn't expired, I would rewrite this. I wrote this in a horribly unclear way.

Re: Unix’s file durability problem

#130

Post by caf »

jstimpfle wrote: post #59
Animats wrote: post #54

I've suggested an approach to this before. There should be several types of files. - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. This should be the default when a file is opened via "creat()" - "Temp" files a…

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

rename(2) atomic replacement only means atomic with respect to processes running on the system at the same time: anyone looking up the newpath will get either the old file or the new file (and won't, for example, see a spurious ENOENT).

It doesn't mean atomic with respect to a system crash.

The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.

Post reply on HN