Live data from Hacker News

Files Are Fraught with Peril

danluu.com

21–30 of 79 posts

Re: Files Are Fraught with Peril

#21
post #17
post #4

This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given that people that specialize in using them can't do it correctly: "Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope would write to files safely, like datbases and version control systems ... they found that eve…

> filesystem API design is obviously a problem It's an API that's so critical it's almost impossible to change or rewrite. The API itself has barely moved in 30 years, whether it's POSIX or Win32. Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.

> Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.

I would love if operating systems exposed a local object storage syscall API (with object versioning and all that good stuff.)

It could be implemented on top of the filesystem for all I care, as long as it's an abstraction with safe object-storage semantics, enforced by the kernel and exposed to all processes without a need for library support.

I believe that processes that wanted to "base" themselves entirely on object-storage would still need to touch the fileystem abstraction as well, mostly to allocate temporary on-disk "buffers" to gradually write to before submitting them to the object-store ABI as new object bodies. But:

1. Most such buffers would be small enough that you could get away with using an anonymous mmap(2) instead, keeping the file "on disk" in the page file rather than in the filesystem itself.

2. For objects you're writing to by streaming, with an unbounded eventual size, you could do the same trick Google Cloud Storage does: allocate a series of fixed-size "chunk" objects to receive the stream data, closing one and opening the next as each previous chunk gets "filled"; and then expose an API call to concatenate such chunk objects together on the object-storage kernel "backend" into single files (probably in O(1) time, because at a low level it's just concatenating disk extent lists.)

3. For other unbounded-size buffers, you could also have the object-store-kernel-daemon provide an API where it manages "large durable working copy" files for you, sort of "checking out" objects into file descriptors (probably using copy-on-write file clones on the backend), then "checking in" file descriptors to become new versions of those objects (maybe even "helpfully" avoiding doing so if the buffer hasn't been touched.)

Re: Files Are Fraught with Peril

#23
Does anyone happen to know if ZFS also suffers from renames being non-atomic? Since ZFS is the only candidate for a reasonable file system we have anyway[+], I'd be a lot less sad if it turns out that way out of this dumpster fire is just tell people to use ZFS.

[+] The absolutely minimal requirement for a reasonable file system is that it works on multiple popular platforms and performs checksumming and other data integrity measures. And that's before we even get to still highly desirable stuff like snaphotting, encryption, etc.

Re: Files Are Fraught with Peril

#24
post #17
post #4

This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given that people that specialize in using them can't do it correctly: "Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope would write to files safely, like datbases and version control systems ... they found that eve…

> filesystem API design is obviously a problem It's an API that's so critical it's almost impossible to change or rewrite. The API itself has barely moved in 30 years, whether it's POSIX or Win32. Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.

The typical solution is to provide a new, safe API while supporting the broken stuff forever.

Re: Files Are Fraught with Peril

#25
post #20

The suggestion to "use SQLite" to write files should come with a caveat. Such files can still become corrupted, and cannot be inspected nor amended with a text editor. Any truly robust file-on-local-disk storage scheme for a non-cloud system should allow for manual diagnostics and repair.

Also, schema updates are quite hopeless with sqlite. One cannot even rename a column last time I checked.

I'm not sure I've ever tried to rename one, but the docs assert support for ALTER TABLE RENAME COLUMN (https://www.sqlite.org/lang_altertable.html).

Re: Files Are Fraught with Peril

#26
post #10
post #5

With the risk of sounding like those /r/programming replies: why would an application be worried about journaling/logs on a file system level? As an application developer all I’m usually told is that the only atomic operations are creates, deletes, and renames. So to update a file you always write a second file and then rename it to the destination. So Copy file.txt to file.txt.new Update file.txt.new Rename file.txt…

This is specifically addressed at one point: > This trick doesn't work. People seem to think that this is safe becaus the POSIX spec says that rename is atomic, but that only means rename is atomic with respect to normal operation, that doesn't mean it's atomic on crash. This isn't just a theoretical problem; if we look at mainstream Linux filesystems, most have at least one mode where rename isn't atomic on crash. R…

A number of applications could be made more reliable if rename was atomic, an acquire barrier, and a release barrier all together.

Re: Files Are Fraught with Peril

#27

Oh god, I didn't realize how broken filesystems are. Shit.

If it helps, just about everything is like that if you look closely. Processors have side channel attacks, RAM has rowhammer which recently turned out to be a real thing, digital electronics in general turn out to have analog side effects, time and space are both basically impossible for computers to represent precisely (see: falsehoods programmers believe about *). We should do what we can, but life goes on:)

Re: Files Are Fraught with Peril

#28
post #8

I did work on backup solutions and supported Dropbox, and I think that the author used Dropbox as a reference to support his own concerns about filesystems that are not related in any case to the Dropbox case. From the filesystem point of view, I don't think that Dropbox is so much concerned about you losing your data because of corruption. As, anyway they are supposed to be safe and versioned in the cloud. And I thi…

Also to note, Dropbox was likely inspired by the open source Unison file-synchronization tool. So they have to handle all of the shortcomings and caveats that Unison does not handle out of the box:

Caveats: https://www.cis.upenn.edu/~bcpierce/unison/download/releases...

Symbolic links: https://www.cis.upenn.edu/~bcpierce/unison/download/releases...

Permissions: https://www.cis.upenn.edu/~bcpierce/unison/download/releases...

Cross platform issues (unsupported characters across operating systems and file systems): https://www.cis.upenn.edu/~bcpierce/unison/download/releases...

Edit: Inspired by the the Unison Spec (published on 2004) which may be why we have conflict files etc. when using Dropbox http://www.cis.upenn.edu/~bcpierce/papers/unisonspec.pdf

Re: Files Are Fraught with Peril

#30
post #8

I did work on backup solutions and supported Dropbox, and I think that the author used Dropbox as a reference to support his own concerns about filesystems that are not related in any case to the Dropbox case. From the filesystem point of view, I don't think that Dropbox is so much concerned about you losing your data because of corruption. As, anyway they are supposed to be safe and versioned in the cloud. And I thi…

Also to note, Dropbox was likely inspired by the open source Unison file-synchronization tool. So they have to handle all of the shortcomings and caveats that Unison does not handle out of the box: Caveats: https://www.cis.upenn.edu/~bcpierce/unison/download/releases... Symbolic links: https://www.cis.upenn.edu/~bcpierce/unison/download/releases... Permissions: https://www.cis.upenn.edu/~bcpierce/unison/download/rele…

Fairly sure this is not true, unless you have a source. Drew did all the client side work in python/librsync to get going.
Post reply on HN