Live data from Hacker News

The fastest rm command and one of the fastest cp commands

alexsaveau.dev

81–83 of 83 posts

Re: The fastest rm command and one of the fastest cp commands

#81

Earlier quoted context omitted.

A directory is a file like anything else that contains a map of names to inodes. If you're trying to add or remove mappings (create or delete files), then clearly some synchronization must occur or the contents of the file will contain garbage. In theory you could get away with a very small critical section that says "lock bytes N through M" of the file, but then how do you deal with disk block alignment (i.e. two pa…

> A directory is a file like anything else that contains a map of names to inodes. If you're trying to add or remove mappings (create or delete files), then clearly some synchronization must occur or the contents of the file will contain garbage. this synchronization is handled for you by the fs, specifically the fs cache inode alignment and errors are managed by this intermediating layer your benchmarks are not demo…

Except they are and your claims are trivial to disprove: simply run the benchmarks under perf. You'll find that most of the time is spent on the rwsem which is described here onwards: https://www.kernel.org/doc/html/latest/filesystems/path-look...

Re: The fastest rm command and one of the fastest cp commands

#82

Earlier quoted context omitted.

> A directory is a file like anything else that contains a map of names to inodes. If you're trying to add or remove mappings (create or delete files), then clearly some synchronization must occur or the contents of the file will contain garbage. this synchronization is handled for you by the fs, specifically the fs cache inode alignment and errors are managed by this intermediating layer your benchmarks are not demo…

Except they are and your claims are trivial to disprove: simply run the benchmarks under perf. You'll find that most of the time is spent on the rwsem which is described here onwards: https://www.kernel.org/doc/html/latest/filesystems/path-look...

your benchmarks use O_DIRECT, which bypasses the fs cache

the fs cache does most/all of the optimizations you're doing manually

bypassing the fs cache is highly atypical for user-space code

Re: The fastest rm command and one of the fastest cp commands

#83

> The key insight is that file operations in separate directories don’t (for the most part) interfere with each other, enabling parallel execution. i'm clearly missing something here parallel execution helps when operations are cpu bound file operations are (almost always) io bound and totally unclear how directories represent an "interference" boundary bizarre

What I would suppose is that it reduces the amount of redundant IO where a directory is edited, flushed to disk, and then later updated again. If all these updates happen in a batch there will be less IO overall.

sure, but the fs cache does all of this kind of stuff for you, right?
Post reply on HN