61–70 of 79 posts
Earlier quoted context omitted.
Firefox reader mode is pretty useful in these cases, and can also save you from websites with the opposite problem.
True, but sadly Chrome still lacks a reader mode for whatever reason
I've been using it for a while and it seems to be solid.
However, I think what motivated Dropbox to drop so many Linux filesystems was the need for a certain flavor of xattrs, but specifically to detect renames, i.e. to merge create/delete events into renames.
Perhaps Dropbox adds internal UIDs via xattrs to every file you add to a Dropbox folder. If you move the file around, no problem, Dropbox can detect the rename via the xattr UID. Relying on heuristics alone to do rename detection can be brittle if you don't do it right, and missing rename events means file version histories get lost, which is a terrible shock for users. Imagine you suddenly can't find last month's version of a file...
So and just guessing here but probably those Linux filesystems lacked performant xattr access and were too slow for Dropbox to do hundreds of thousands of xattr lookups when scanning a folder at startup or every N minutes, or whenever inotify tells them something changed.
They didn't want the complexity of heuristics, or they found a way to exploit EXT on-disk data structures for faster xattr lookups, something like that. Who knows? Perhaps the real reason is less complicated than that.
Earlier quoted context omitted.
That's where the "and versioned" comes in handy
Versioned wont help if it's the first copy that got into dropbox.
Earlier quoted context omitted.
Text-based formats are the best for repairability but not so efficient for structuring, querying and storing information. It’s a tradeoff, as always!
A search index can be maintained in binary format; it can be rebuilt in the event of corruption.
- An index is redundant information but dropping it and recreating it is not “repairability” as in “I read and amend a text”. You compare apples to oranges.
- What I wrote still applies to an index, so that’s orthogonal: a binary index will be more efficient but harder to repair than a text-based binary index.
This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given that people that specialize in using them can't do it correctly: "Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope would write to files safely, like datbases and version control systems ... they found that eve…
I write database engines for Linux and the filesystem situation really is a train wreck. It isn't anyone's fault per se, design choices and standards were accreted over many decades that in isolation made sense in some context but which in aggregate have many poorly defined interactions and create conflicting requirements. And you can't change any of it easily because there are several decades of software built using…
I was also (incorrectly) assuming that renaming a file after having fsynced the contents is a valid mechanism to perform "atomic" file writes where the write either appears completely at the destination path, creates a zero byte file or does not happen at all. But it should never produce an "incorrect" non-zero byte file - even in the face of crashes.
It is definitely something that is used in the wild so it was surprising to me that it is not correct. I even found a recent LWN article suggesting otherwise, so there seems to be a lot of confusion on this topic.
Would love to learn more if anybody has detailed infos or a link to the relevant mailing list thread or similar.
EDIT: Glibc documentation also implies that the "rename trick" is indeed safe:
From https://www.gnu.org/software/libc/manual/html_node/Renaming-...
> One useful feature of rename is that the meaning of newname changes “atomically” from any previously existing file by that name to its new meaning (i.e., the file that was called oldname). There is no instant at which newname is non-existent “in between” the old meaning and the new meaning. If there is a system crash during the operation, it is possible for both names to still exist; but newname will always be intact if it exists at all.
Another relevant source that suggests the trick is safe is https://lwn.net/Articles/327601/. Keep in mind that Ted Tso is the ext4 maintainer.
> For the longer term, Ted asked: should the above-described fixes become a part of the filesystem policy for Linux? In other words, should application developers be assured that they'll be able to write a file, rename it on top of another file, omit fsync(), and not encounter zero-length files after a crash? The answer turns out to be "yes," but first Ted presented his other long-term ideas.
> If we look at a worn out drive, one very close to end-of-life, it's specced to retain data for one year to three months, depending on the class of drive. What happens once the data expires? Does the SSD return an error when the data is read, or does it read the bogus data without knowing? I would kind of prefer the drive bricking itself rather than risking silently backing up bogus data. The earlier comment about E…
Earlier quoted context omitted.
I write database engines for Linux and the filesystem situation really is a train wreck. It isn't anyone's fault per se, design choices and standards were accreted over many decades that in isolation made sense in some context but which in aggregate have many poorly defined interactions and create conflicting requirements. And you can't change any of it easily because there are several decades of software built using…
I'm curious, what low-level storage interfaces do you recommend?
Does anyone happen to know if ZFS also suffers from renames being non-atomic? Since ZFS is the only candidate for a reasonable file system we have anyway[+], I'd be a lot less sad if it turns out that way out of this dumpster fire is just tell people to use ZFS. [+] The absolutely minimal requirement for a reasonable file system is that it works on multiple popular platforms and performs checksumming and other data i…
ZFS never reorders metadata operations in an observable manner, so it's really well behaved. If only every filesystem was like that... You still need fsync for the data, but you can set sync=disabled on the filesystem, which turns it into a barrier. Alas, you can't do anything more granular than per-filesystem.
def atomically_write(filename, data):
with open(filename + '.tmp', 'w') as fh:
fh.write(data)
os.rename(filename + '.tmp', filename)
> If only every filesystem was like that..
I wonder if the solution for databases and other server software that needs to persist data reliably is just pretend they are and insist on end-users sticking to non-broken filesystems (most probably ZFS, assuming the set is currently non-empty). The alternative of wasting endless time and resources on sysiphean quests to placate crap filesystems seems to have been an utter failure so far.