Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to rewrite files in Linux

bugs.edge.launchpad.net

Post reply on HN
Print view View original

11–20 of 22 posts

Re: How to rewrite files in Linux

#11

Post by utx00 »

swillden wrote: post #9

Another option: use mmap() and rewrite it in place, using ftruncate() if necessary to grow/shrink the file, and msync() to flush the changes. This is also risky because it could leave the file in an inconsistent, partially-modified state. However, in practice it seems to work really well, and Google turns up a striking lack of articles discussing the risks of inconsistency due to use of mmap. I'd like to learn more a…

might using a berkdb (as suggested) make more sense for your case? it can be devilish tricky getting these things to work right.

Re: How to rewrite files in Linux

#12

Post by tptacek »

jws wrote: post #10

guilty of #2. I'll spend part of this afternoon putting in fsyncs(). Pseudo related: I wish I could have anonymous temporary files and associate them with a name later. Temporary name handling is just bugs waiting to happen. Even if you do it securely, how many programmers have a way of cleaning up spurious debris that might be left after an inelegant termination?

You create secure temp files with mkstemp(). It's built into the standard C library.

Mad respect to Ts'o, but most Unix apps --- regardless of what he thinks of the developers --- don't do this fsync dance. There isn't an epidemic of corrupted files because of it. And there have been notable cases where syncing has caused calamitous performance problems; for instance, look up what the Apple developers did to sqlite.

Re: How to rewrite files in Linux

#13

Post by jws »

jws wrote: post #10

guilty of #2. I'll spend part of this afternoon putting in fsyncs(). Pseudo related: I wish I could have anonymous temporary files and associate them with a name later. Temporary name handling is just bugs waiting to happen. Even if you do it securely, how many programmers have a way of cleaning up spurious debris that might be left after an inelegant termination?

The first application I went to fix is in PHP, there does not seem to be an fsync() call in PHP. I may not succeed this afternoon.

Re: How to rewrite files in Linux

#14

Post by swillden »

utx00 wrote: post #11
swillden wrote: post #9

Another option: use mmap() and rewrite it in place, using ftruncate() if necessary to grow/shrink the file, and msync() to flush the changes. This is also risky because it could leave the file in an inconsistent, partially-modified state. However, in practice it seems to work really well, and Google turns up a striking lack of articles discussing the risks of inconsistency due to use of mmap. I'd like to learn more a…

might using a berkdb (as suggested) make more sense for your case? it can be devilish tricky getting these things to work right.

Possibly. The ability to view/tweak the file in a text editor is something I find very convenient, but it's less important than consistency.

Re: How to rewrite files in Linux

#15

Post by wmf »

pmjordan wrote: post #4
wmf wrote: post #3

For approach #2, can you lose the file completely or only the new version? In many cases it would be acceptable if the old version was still there after a crash (ACI semantics).

As I understand it, you can lose both versions in #2. The rename drops the refcount of the old file to 0, freeing its inode and associated file blocks. If you're lucky, they haven't been overwritten and can still find them. Not easily from an app though. On the other hand, the new file might not have been flushed to disk yet. I suspect this could be solved fs-side using a journal that doesn't just store metadata (as…

Ugh, that is bad if true. A filesystem shouldn't commit a rename to disk before the file being renamed; that is common sense causality. Especially since (AFAIK) #2 is totally safe under ZFS, so Linux is behind yet again.

Re: How to rewrite files in Linux

#17

Post by jrockway »

swillden wrote: post #14
utx00 wrote: post #11

Earlier quoted context omitted.

might using a berkdb (as suggested) make more sense for your case? it can be devilish tricky getting these things to work right.

Possibly. The ability to view/tweak the file in a text editor is something I find very convenient, but it's less important than consistency.

You can always write a wrapper to start a transaction, extract the file from the database, exec $EDITOR on it, replace the database data with the file, and commit the transaction. That is very safe, and very easy to do. (And, you can make the txn apply to multiple files, which is quite useful.)

We use this technique in the command line utility for KiokuDB.

Re: How to rewrite files in Linux

#18

Post by nkurz »

etal wrote: post #8

The whole bug thread is good information, but: if application developers actually follow approach #3 en masse, won't that massively slow down the system? I see some cargo-cult potential in fsync-after-every-write here. The emacs example may be correct, but it has a couple steps that require extra thought depending on the context of the rewrite. Where data consistency and security are less necessary, example #2 seems…

"example #2 seems just fine, assuming rename() is atomic and a power failure would just leave the old contents in the file."

Unfortunately, the 'bug' is that this does not happen. Instead, the truncate hits disk before the new data is written, so that a power loss leaves the file empty rather than merely out-of-date. This behaviour is different on ext4 than on ext3, hence it has only recently become an issue.

To my untrained eye, regardless of what the POSIX standard 'allows', this is a bug that should be fixed. On the bright side, it looks like there are patches ready to be applied that will change the behaviour to match ext3, which is as you describe. If this happens, #2 would remain a relatively safe choice.

Re: How to rewrite files in Linux

#19

Post by nkurz »

swillden wrote: post #9

Another option: use mmap() and rewrite it in place, using ftruncate() if necessary to grow/shrink the file, and msync() to flush the changes. This is also risky because it could leave the file in an inconsistent, partially-modified state. However, in practice it seems to work really well, and Google turns up a striking lack of articles discussing the risks of inconsistency due to use of mmap. I'd like to learn more a…

"Another option: use mmap() and rewrite it in place, using ftruncate() if necessary to grow/shrink the file, and msync() to flush the changes."

This is a fine option, but for the purpose of this bug I don't think there is any advantage over using stdio (with seeks if necessary) and calling fsync() after writing. It's a great technique, but functionally equivalent to just saying "avoid O_TRUNCATE".

Unless you have some nifty way of upgrading from a copy-on-write map(MAP_PRIVATE) to something that gets atomically written to disk?

Post reply on HN