Mtime comparison considered harmful (2018) • Hacker News Board

Re: Mtime comparison considered harmful (2018)

#31

Post by lathiat » Tue, Sep 13, 2022, 3:53 PM UTC

dezgeg wrote: post #30

It would be neat if filesystems like btrfs or ZFS could expose their internal checksum of the file data to userspace, to quickly see if two files are having identical content (eg. think of rsync). (Assuming the hashes computed internally the filesystem are actually purely based on the file data, and not for example metadata like block pointers. But, dedup-capable filesystems should surely have such checksum internall…

The problem with that is for good performance reasons that checksum is typically on smaller blocks than the whole file. For example ZFS splits files into a maximum of “recordsize” (128K) by default and checksums each block.

Not so likely whatever remote target you are looking at has the same blocks and checksum algorithm.

On the upside you can often ask such file systems to take two snapshots and what changed between them. Or to export some kind of differential to transform the original snapshot to the newer snapshot. Both ZFS and Btrfs can do those.

Re: Mtime comparison considered harmful (2018)

#32

Post by Karellen » Tue, Sep 13, 2022, 9:26 PM UTC

dezgeg wrote: post #29

Karellen wrote: post #26

> Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures. You normally don't see any symptoms of this except that hardlinks are suspiciously slow on MacOS. But in exchange for the slowness, the kernel actually can look up all filenames of a hardlink if it wants. I think this has something to do with Aliases and…

I suppose it's perfectly allowed for MacOS to simulate Unix hard link semantics (ie. names are just pointers to inodes, with each name being equal) even though the underlying filesystem doesn't have the name+inode split. It seems indeed this is the case on HFS+: https://developer.apple.com/library/archive/technotes/tn/tn1...

Ah, that makes it much clearer, thanks.

Re: Mtime comparison considered harmful (2018)

#33

Post by 95014_refugee » Tue, Sep 13, 2022, 10:03 PM UTC

Karellen wrote: post #26

> Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures. You normally don't see any symptoms of this except that hardlinks are suspiciously slow on MacOS. But in exchange for the slowness, the kernel actually can look up all filenames of a hardlink if it wants. I think this has something to do with Aliases and…

It reads weird because it's a roughly equal mix of opinion and wrong, possibly informed by some historical understanding of how HFS+ used to work.

Granted, in 2018 HFS+ was still kind of relevant, but even at that point the listed assertions were pretty dodgy.

Re: Mtime comparison considered harmful (2018)

#34

Post by rleigh » Wed, Sep 14, 2022, 9:36 AM UTC

lathiat wrote: post #31

dezgeg wrote: post #30

It would be neat if filesystems like btrfs or ZFS could expose their internal checksum of the file data to userspace, to quickly see if two files are having identical content (eg. think of rsync). (Assuming the hashes computed internally the filesystem are actually purely based on the file data, and not for example metadata like block pointers. But, dedup-capable filesystems should surely have such checksum internall…

The problem with that is for good performance reasons that checksum is typically on smaller blocks than the whole file. For example ZFS splits files into a maximum of “recordsize” (128K) by default and checksums each block. Not so likely whatever remote target you are looking at has the same blocks and checksum algorithm. On the upside you can often ask such file systems to take two snapshots and what changed between…

For ZFS, we have the pool-wide transaction number, which increments with every disk write. Couldn't you use that as the "time" for the last modification to each file so long as you have a starting transaction number to compare it against?

Re: Mtime comparison considered harmful (2018)

#35

Post by osigurdson » Wed, Sep 14, 2022, 7:56 PM UTC

Someone wrote: post #14

osigurdson wrote: post #3

The title should be: “I don’t like Mtime comparison. Here’s why.” If your name is Dijkstra and it’s 1968 feel free to use the phrase “considered harmful”, otherwise use plain language.

Nitpick: Dijkstra didn’t use “considered harmful”, Niklaus Wirth did. https://en.wikipedia.org/wiki/Considered_harmful#History : The original title of the letter, as submitted to CACM, was "A Case Against the Goto Statement", but CACM editor Niklaus Wirth changed the title to "Goto Statement Considered Harmful". Regarding this new title, Donald Knuth quipped that "Dr. Goto cheerfully complained that he was always bei…

Noted.

Re: Mtime comparison considered harmful (2018)

#36

Post by Doxin » Fri, Sep 16, 2022, 11:28 AM UTC

junon wrote: post #27

photon-torpedo wrote: post #19

Earlier quoted context omitted.

The author mentions that they also wrote a backup program (bup), and for backup programs it would be very convenient if directory mtimes would get updated like this (recursively up to the root), as it would allow to skip scanning the entire filesystem for changed files (which in my experience is where backup programs spend most of their time).

I don't remember if mtime is updated on each write call or just on fopen but I could see this being a huge performance overhead, namely for applications that are FS bound. I wonder if io_uring would help the situation though since it's mainly geared toward filesystem operations.

No reason a hypothetical recursive mtime needs to be atomic. The kernel could just stick it in a buffer somewhere and deal with that sort of think out-of-band and in batches. You'd probably need some filesystem journaling trickery if you want to make sure the recursive mtime always updates eventually when a file is modified.