Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

111–120 of 161 posts

Re: Unix’s file durability problem

#111

Post by jstimpfle »

Animats wrote: post #67
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

> On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position.

I would never expect that. It's not how it works. But what has to work is the dup() case, or the inherited descriptors case like

$ ( do_this & do_that & } > /some/file

because here they share the same file descriptor (as opposed to having only the same file opened on stdout). Note this still doesn't guarantee that writes won't be interleaved. I think there is only a guarantee that writes up to 512 bytes won't be interleaved. The answer here is: Don't do simultaneous writes. Or implement your own synchronization mechanism. It's just not a requirement with a universally satisfying solution that should be implemented in the kernel.

> NTFS doesn't do append correctly.

If you meant NFS: It deliberately violates POSIX file system semantics. I guess it's just too hard to implement. For example, an NFS server can't know whether the machine which owned a lock crashed or is just temporarily disconnected from the network (and has still a process running thinking it owns a lock and not knowing the server is wondering).

In any case, not the API to blame.

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.

How could we possibly implement these "transactions" if the write is _not_ at the end of some file?

> It's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.

What the disk does is in the end controlled only by the disk itself (and of course environmental forces). So it's always best effort.

Re: Unix’s file durability problem

#112

Post by avar »

haberman wrote: post #105
avar wrote: post #99

Earlier quoted context omitted.

I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF? The problem you're describing is basically solved on modern POSIX OS's. You're just proposing moving the semantics of what could be better done in kernel-space by resizing PIPE_BUF to userspace. The way this works now is that you open a logfile with O_APPEND, any write(2) you make to the file will be atomic up to PIPE_…

> I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF? Nope. I was replying to one proposed semantic and arguing that the ideal semantic is slightly different. I haven't looked deeply into how such a semantic could or could not be mapped onto current APIs. > at the point where you're making >4K writes you're usually better off having some custom log system anyway and not…

    > I just read about O_APPEND and it doesn't sound like what I'm
    > talking about at all. It always appends to the end. It sounds like
    > if two writers race, both chunks of new data are appended. That's
    > not as useful as what I described. I'm basically talking about
    > compare and swap, except instead of compare and swap its compare and
    > append
Correct. If you want to only write to the end of a file if that file is size X you'd either need your proposed syscall of "only write if size is X" or have every writer cooperate via flock() to achive that via current semantics.

That's not at all what the grandparent is talking about though. They just want "log" files where you can only append and the write() is atomic. That doesn't mean "my write shouldn't work if there is a concurrent write" which is a semantic you're making up, and which I don't really see making sense for log files.

I'm really finding it difficult to imagine a plausable use case for your proposed semantics. Imagine this: You have 100 concurrent web serving processes all writing to an access.log, currently they can just write() O_APPEND to an access.log with a string below PIPE_BUF and their writes will end up on disk, but may not be interleaved in "real time", but who cares?

What you're proposing means that only 1 out of those 100 processes will succeed in their write() call. The rest will start a retry loop that's trying to write out some data whose order didn't even matter in the first place to try to flush it to disk because they all want to only write the data if the size of the file is N, and every single write updates N.

If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files, and the way it's done is not to try to add a new API to the kernel, you just send messages to some userspace thread that queues them up and does the I/O for you.

Re: Unix’s file durability problem

#113

Post by kentonv »

jstimpfle wrote: post #103
ryao wrote: post #89

Earlier quoted context omitted.

ZFS does do this. Changes are atomic, so they either happen or do not. There is no in-between.

I don't know about ZFS, but the usual write(2) API does not support this: It might for example return early with a short write because some interrupt occurred. Can happen for all "slow devices" (see signal(7) ). And I think that's a good thing and am sure many programs expect this.

The signal(7) man page also states clearly that a local disk is not a "slow" device, so this seems moot.

    read(2), readv(2), write(2), writev(2), and ioctl(2) calls on "slow" devices.  A "slow" device is one where the I/O call  may  block  for  an
    indefinite  time, for example, a terminal, pipe, or socket.  If an I/O call on a slow device has already transferred some data by the time it
    is interrupted by a signal handler, then the call will return a success status (normally, the number of  bytes  transferred).   Note  that  a
    (local) disk is not a slow device according to this definition; I/O operations on disk devices are not interrupted by signals.

Re: Unix’s file durability problem

#114

Post by avar »

twa927 wrote: post #44
rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

> A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen(). So if I need to create a file with a timestamp, you are suggesting creating a SQLite database, a table, and inserting my timestamp there? It's nonsensical to "fix" a Unix low-level syscall issue by using SQLite.

I'll bite. Please post the C code required to portably and safely create a file and write a timestamp to it in a way that beats the safety provided by SQLite.

Most C programmers can't even write that piece of code. Let's see if you can.

Re: Unix’s file durability problem

#115

Post by colanderman »

anarazel wrote: post #95
colanderman wrote: post #75

Earlier quoted context omitted.

sync_file_range(2) allows you to specify ranges to sync, instead of the whole file, but it's not that useful anyway. In most DBs, you have separate data and log. The log is always only appended to, and the nature of disks is such that grouping concurrent transactions together into a single commit is a big win regardless of how syncing works. So syncing all outstanding I/O to the log file is generally what you want to…

Actually, we made postgres use sync_file_range() for checkpointing if available in 9.6. Not for durability - there's a few to many caveats in the manpage - but to control how much work a later fsync() has to do. In many workloads checkpointing can generate a lot of writes, and the OS's writeback caching of those can generate a lot of dirty buffers in the kernel's page cache. If the kernel decides to flush those (on i…

Thanks for the info, as a heavy user of Postgres it's great to know that option will be available!

Re: Unix’s file durability problem

#116

Post by jstimpfle »

kentonv wrote: post #113
jstimpfle wrote: post #103

Earlier quoted context omitted.

I don't know about ZFS, but the usual write(2) API does not support this: It might for example return early with a short write because some interrupt occurred. Can happen for all "slow devices" (see signal(7) ). And I think that's a good thing and am sure many programs expect this.

The signal(7) man page also states clearly that a local disk is not a "slow" device, so this seems moot. read(2), readv(2), write(2), writev(2), and ioctl(2) calls on "slow" devices. A "slow" device is one where the I/O call may block for an indefinite time, for example, a terminal, pipe, or socket. If an I/O call on a slow device has already transferred some data by the time it is interrupted by a signal handler, th…

oops, had that wrong. Thanks for noticing!

Re: Unix’s file durability problem

#117

Post by haberman »

avar wrote: post #112
haberman wrote: post #105

Earlier quoted context omitted.

> I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF? Nope. I was replying to one proposed semantic and arguing that the ideal semantic is slightly different. I haven't looked deeply into how such a semantic could or could not be mapped onto current APIs. > at the point where you're making >4K writes you're usually better off having some custom log system anyway and not…

> I just read about O_APPEND and it doesn't sound like what I'm > talking about at all. It always appends to the end. It sounds like > if two writers race, both chunks of new data are appended. That's > not as useful as what I described. I'm basically talking about > compare and swap, except instead of compare and swap its compare and > append Correct. If you want to only write to the end of a file if that file is si…

> You have 100 concurrent web serving processes all writing to an access.log

That makes no sense. Have each process write to its own log. This is making your life hard for no reason.

> If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files

This isn't about access.log, it's about transaction logs for database systems. Like the commit log for Postgres or SQLite. You can think of their commit logs as like a block chain. Imagine a block chain that didn't have a pointer to the previous block, but just a timestamp. It wouldn't work. If you have two conflicting blocks you need some central arbiter to decide which wins.

Re: Unix’s file durability problem

#118

Post by colanderman »

ryao wrote: post #97
colanderman wrote: post #65

Earlier quoted context omitted.

Don't know why you're downvoted. Everything you said is correct. Additionally, in Linux, you can get asynchronous fdatasync(2) with sync_file_range(2). aio(7) is weird. It really only works with direct I/O (which is necessary much less often than people think it is), and IIRC aio_fsync(2) isn't implemented on Linux, but that doesn't matter so much because generally direct I/O implies synchronous I/O (but not always).…

AIO does not require O_DIRECT. Most Linux file systems do not implement aio_fsync because they put the writes into the page cache and block until it is flushed to disk before signaling completion. That means aio_fsync can be implemented by no-op.

Unfortunately that is not documented in any man page I've read, because that is very useful (crucial) to know and not at all obvious. I reiterate that aio is weird :)

Also, I'm pretty sure that issuing IOCB_CMD_FSYNC from io_submit(2) has given me an error in the past. Not sure about aio_fsync(2), because last time I checked, POSIX aio only had a pthread-based backend, rather than the native support that libaio (i.e. io_submit(2)) provides.

Re: Unix’s file durability problem

#119

Post by Analemma_ »

asveikau wrote: post #109
Animats wrote: post #67

Earlier quoted context omitted.

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

> NTFS doesn't do append correctly. Care to elaborate on that? I haven't seen issues with the two mechanisms I know of in NT: 1. CreateFile with FILE_APPEND_DATA 2. WriteFile with the OVERLAPPED offset set to ~0 Both of these get you writes that are as far as I know safe to have multiple processes writing to a common log file. Not sure about flushes to disk but you could probably get away with FlushFileBuffers for th…

NTFS handles append just fine. As another comment speculated, the parent may have meant NFS, which does notoriously have lots of problems with append, and many other POSIX semantics.

Re: Unix’s file durability problem

#120

Post by avar »

haberman wrote: post #117
avar wrote: post #112

Earlier quoted context omitted.

> I just read about O_APPEND and it doesn't sound like what I'm > talking about at all. It always appends to the end. It sounds like > if two writers race, both chunks of new data are appended. That's > not as useful as what I described. I'm basically talking about > compare and swap, except instead of compare and swap its compare and > append Correct. If you want to only write to the end of a file if that file is si…

> You have 100 concurrent web serving processes all writing to an access.log That makes no sense. Have each process write to its own log. This is making your life hard for no reason. > If you have a use-case for strictly serializing writes to disk like this that's fine, but it's not the common case with log files This isn't about access.log, it's about transaction logs for database systems. Like the commit log for Po…

    > That makes no sense. Have each process write to its own log. This is
    > making your life hard for no reason.
This is literally how pretty much every daemon process works on Unix systems. You don't log in and tail /var/log/apache/access.log.{1..N} where N is the number of threads/processes. You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics.

Having to tail N logs where N is some unknown number of processes is what's making your life harder for no reason. That's what O_APPEND and PIPE_BUF are for.

    > This isn't about access.log, it's about transaction logs for
    > database systems. [...] think of their commit logs as like a
    > block chain.
I really don't see how any of this is relevant to the original point of having "log" files. Now you're talking about DB transaction logs and blockchains, which really don't need to have their nuances implemented in terms of POSIX file semantics. They can just have a single process that manages those expectations.

Everyone's expectations don't have to be implemented on the filesystem level, particularly when those expectations inherently involve locking writes and retry loops.

Post reply on HN