Oh god, I didn't realize how broken filesystems are. Shit.
If it helps, just about everything is like that if you look closely. Processors have side channel attacks, RAM has rowhammer which recently turned out to be a real thing, digital electronics in general turn out to have analog side effects, time and space are both basically impossible for computers to represent precisely (see: falsehoods programmers believe about *). We should do what we can, but life goes on:)
Files Are Fraught with Peril
31–40 of 79 posts
Re: Files Are Fraught with Peril
#32> If we look at a worn out drive, one very close to end-of-life, it's specced to retain data for one year to three months, depending on the class of drive.
What happens once the data expires? Does the SSD return an error when the data is read, or does it read the bogus data without knowing?
I would kind of prefer the drive bricking itself rather than risking silently backing up bogus data. The earlier comment about ECC and bit error rates suggests bad reads are identified as such, but I'm not sure how far to trust that given that, as mentioned, I/O is hard.
Re: Files Are Fraught with Peril
#33This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given that people that specialize in using them can't do it correctly: "Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope would write to files safely, like datbases and version control systems ... they found that eve…
I write database engines for Linux and the filesystem situation really is a train wreck. It isn't anyone's fault per se, design choices and standards were accreted over many decades that in isolation made sense in some context but which in aggregate have many poorly defined interactions and create conflicting requirements. And you can't change any of it easily because there are several decades of software built using the existing design. Guaranteeing precise, consistent behavior is nearly impossible with the standard filesystem API infrastructure and the implementation details change invisibly in important ways.
This is evident in database engines if you look at the number of lines of code dedicated to storage. Working with raw block devices requires the fewest lines of code, working through the filesystem requires the most, and direct I/O (partial filesystem bypass) is somewhere in the middle. And even if you design for comparable guarantees across these models for working with storage, the consistency of behavior across Linux environments also significantly improves as you bypass more of the filesystem.
Ironically, I now tend to borrow the low-level storage interface from database kernels, which abstracts the filesystem mess, for all code that needs to work with storage even if it isn't a database. It provides a saner interface with more consistent guarantees and often better performance. In my ideal world, someone properly designs a completely new filesystem API from scratch that sits alongside the legacy APIs that applications could start migrating to. But it would probably require adverse changes in the way the Linux kernel works for the legacy path, which means it will never happen.
Re: Files Are Fraught with Peril
#34Earlier quoted context omitted.
Also to note, Dropbox was likely inspired by the open source Unison file-synchronization tool. So they have to handle all of the shortcomings and caveats that Unison does not handle out of the box: Caveats: https://www.cis.upenn.edu/~bcpierce/unison/download/releases... Symbolic links: https://www.cis.upenn.edu/~bcpierce/unison/download/releases... Permissions: https://www.cis.upenn.edu/~bcpierce/unison/download/rele…
Fairly sure this is not true, unless you have a source. Drew did all the client side work in python/librsync to get going.
Ok, may have been inspired by. Here's some relevant backstory from 2011:
https://www.wired.com/2011/12/backdrop-dropbox/
Re: Files Are Fraught with Peril
#35Does anyone happen to know if ZFS also suffers from renames being non-atomic? Since ZFS is the only candidate for a reasonable file system we have anyway[+], I'd be a lot less sad if it turns out that way out of this dumpster fire is just tell people to use ZFS. [+] The absolutely minimal requirement for a reasonable file system is that it works on multiple popular platforms and performs checksumming and other data i…
I love ZFS but the licence situation means it'll always be a mess to use.
Re: Files Are Fraught with Peril
#36> [ext4 data= mount option] writeback: Data ordering is not preserved – data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest-throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.
Not just old data from the same file, either. In the case of file appending, "old" can actually mean "whatever junk happened to be in that allocated disk block earlier".
Re: Files Are Fraught with Peril
#37Earlier quoted context omitted.
> filesystem API design is obviously a problem It's an API that's so critical it's almost impossible to change or rewrite. The API itself has barely moved in 30 years, whether it's POSIX or Win32. Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.
> Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native. I would love if operating systems exposed a local object storage syscall API (with object versioning and all that good stuff.) It could be implemented on top of the filesystem for all I care, as long as it's an abstraction with safe…
What's the difference between an 'object' and a file?
Re: Files Are Fraught with Peril
#38Oh god, I didn't realize how broken filesystems are. Shit.
If it helps, just about everything is like that if you look closely. Processors have side channel attacks, RAM has rowhammer which recently turned out to be a real thing, digital electronics in general turn out to have analog side effects, time and space are both basically impossible for computers to represent precisely (see: falsehoods programmers believe about *). We should do what we can, but life goes on:)
I'm amazed every time a machine boots.
Given the thousands of things that could cause it not to it's incredible it makes it down the happy path each time.
Re: Files Are Fraught with Peril
#39With the risk of sounding like those /r/programming replies: why would an application be worried about journaling/logs on a file system level? As an application developer all I’m usually told is that the only atomic operations are creates, deletes, and renames. So to update a file you always write a second file and then rename it to the destination. So Copy file.txt to file.txt.new Update file.txt.new Rename file.txt…
Incidentally the copy/rename process can run into issues on Windows due to Windows Defender holding a lock on the file which can prevent it being renamed. This is an issue for the Rust updater utility: https://github.com/rust-lang/rustup.rs/issues/1436
File locks and virus checkers. Two things that I'm not missing since moving from Windows.
Re: Files Are Fraught with Peril
#40Does anyone happen to know if ZFS also suffers from renames being non-atomic? Since ZFS is the only candidate for a reasonable file system we have anyway[+], I'd be a lot less sad if it turns out that way out of this dumpster fire is just tell people to use ZFS. [+] The absolutely minimal requirement for a reasonable file system is that it works on multiple popular platforms and performs checksumming and other data i…
ZFS never reorders metadata operations in an observable manner, so it's really well behaved. If only every filesystem was like that...
You still need fsync for the data, but you can set sync=disabled on the filesystem, which turns it into a barrier. Alas, you can't do anything more granular than per-filesystem.