How to rewrite files in Linux

#1

Post by kqr2 » Sun, Mar 08, 2009, 6:50 AM UTC

How to rewrite files in Linux
bugs.edge.launchpad.net

Re: How to rewrite files in Linux

#2

Post by blasdel » Sun, Mar 08, 2009, 7:51 AM UTC

Written by Ted Ts'o, the author of ext4 (among other things), who's been a linux kernel developer since the very beginning.

Even more approaches to rewriting: http://bitworking.org/news/390/text-editor-saving-routines

Re: How to rewrite files in Linux

#3

Post by wmf » Sun, Mar 08, 2009, 8:38 AM UTC

For approach #2, can you lose the file completely or only the new version? In many cases it would be acceptable if the old version was still there after a crash (ACI semantics).

Re: How to rewrite files in Linux

#4

Post by pmjordan » Sun, Mar 08, 2009, 11:37 AM UTC

wmf wrote: post #3

For approach #2, can you lose the file completely or only the new version? In many cases it would be acceptable if the old version was still there after a crash (ACI semantics).

As I understand it, you can lose both versions in #2. The rename drops the refcount of the old file to 0, freeing its inode and associated file blocks. If you're lucky, they haven't been overwritten and can still find them. Not easily from an app though. On the other hand, the new file might not have been flushed to disk yet.

I suspect this could be solved fs-side using a journal that doesn't just store metadata (as most do) but also file content. Slow, though, as files are effectively written to disk twice.

Re: How to rewrite files in Linux

#5

Post by glymor » Sun, Mar 08, 2009, 11:48 AM UTC

wmf wrote: post #3

For approach #2, can you lose the file completely or only the new version? In many cases it would be acceptable if the old version was still there after a crash (ACI semantics).

My guess is the rename must occur immediately or he would have covered it.

But even if it doesn't the write and the rename are not correlated so it's still not safe. e.g. the move does not occur iff the write completes.

Is there any way to queue the move as a system call?

Re: How to rewrite files in Linux

#6

Post by zafarali » Sun, Mar 08, 2009, 1:02 PM UTC

cool

Re: How to rewrite files in Linux

#7

Post by moe » Sun, Mar 08, 2009, 2:56 PM UTC

I like the robot analogy.

Applies to other programming problems as well, I think I'll use this when I next time see someone "optimizing" SQL database transactions.

Re: How to rewrite files in Linux

#8

Post by etal » Sun, Mar 08, 2009, 3:22 PM UTC

The whole bug thread is good information, but: if application developers actually follow approach #3 en masse, won't that massively slow down the system? I see some cargo-cult potential in fsync-after-every-write here. The emacs example may be correct, but it has a couple steps that require extra thought depending on the context of the rewrite. Where data consistency and security are less necessary, example #2 seems just fine, assuming rename() is atomic and a power failure would just leave the old contents in the file.

For Gnome and KDE, Ted suggests using BDB or an idealized version of SQLite instead of the current dot-directories, but I don't imagine that change being implemented any time soon -- Firefox still has some lingering problems from that transition.

Re: How to rewrite files in Linux

#9

Post by swillden » Sun, Mar 08, 2009, 3:52 PM UTC

Another option: use mmap() and rewrite it in place, using ftruncate() if necessary to grow/shrink the file, and msync() to flush the changes.

This is also risky because it could leave the file in an inconsistent, partially-modified state. However, in practice it seems to work really well, and Google turns up a striking lack of articles discussing the risks of inconsistency due to use of mmap.

I'd like to learn more about the risks of this approach, since I'm using it to selectively rewrite portions of a very important file in a project of mine. I'm pretty sure that my usage pattern is safe. The writes are small enough that they nearly always affect only one disk block and never touch more than two, and I'm careful to msync() at the right places (actually mmap.mmap.flush(), since I'm using Python), but I'm interested in learning about potential issues. This file is too large to make copying it convenient, but I'll do that if necessary.

Re: How to rewrite files in Linux

#10

Post by jws » Sun, Mar 08, 2009, 4:41 PM UTC

guilty of #2. I'll spend part of this afternoon putting in fsyncs().

Pseudo related: I wish I could have anonymous temporary files and associate them with a name later. Temporary name handling is just bugs waiting to happen. Even if you do it securely, how many programmers have a way of cleaning up spurious debris that might be left after an inelegant termination?