Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fastest rm command and one of the fastest cp commands

alexsaveau.dev

Post reply on HN
Print view View original

11–20 of 83 posts

Re: The fastest rm command and one of the fastest cp commands

#11

Post by eru »

saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

Re: The fastest rm command and one of the fastest cp commands

#12

Post by eru »

guiand wrote: post #2

This could be useful for e.g. bazel, where I’ve regularly seen deleting the bazel caches take on the order of 10 minutes because of absurdly large and repeated directory trees (caused by things like runfile trees containing the python interpreter).

The lazy delete as outlined in https://news.ycombinator.com/item?id=35309328 should also work?

Re: The fastest rm command and one of the fastest cp commands

#13

Post by adr1an »

tedunangst wrote: post #5

If you need a large sub tree gone, mv dir .old-dir && rm -r .old-dir & works pretty well. Faster even than an optimized parallel unlinker.

Exactly, I use 'gio trash' on Linux for this reason. And I let a cronjob do removal of all the .Trash with find -ctime +180 -exec rm...

Re: The fastest rm command and one of the fastest cp commands

#14

Post by oefrha »

eru wrote: post #11
saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

They develop https://github.com/saagarjha/unxip, “a fast Xcode unarchiver”. Very few people should routinely delete Xcode.

Re: The fastest rm command and one of the fastest cp commands

#15

Post by LeoPanthera »

eru wrote: post #11
saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

Not OP, but disk space maybe? It's pretty big.

$ du -Ash Xcode.app

22G Xcode.app

Re: The fastest rm command and one of the fastest cp commands

#16

Post by HeavyFeather »

eyelidlessness wrote: post #9
bee_rider wrote: post #6

Earlier quoted context omitted.

That’s pretty outside-the-box and fun.

Amusingly it’s basically how most GUI desktops “delete” stuff, they just put a confirmation in place of the &&

I don't think any GUI move items before a confirmation, so I don't see the link at all.

Re: The fastest rm command and one of the fastest cp commands

#17

Post by zamalek »

rektide wrote: post #3

I cant wait to see the future where we have uring (io_uring) based tools that all work async, getting hopefully embarassingly parallel. Heck yes copying & deleting 2x as fast. Actually I'm on btrfs so reflink copy is like instant I think? I should test that better.

async nearly always trades concurrency for latency, so it wouldn't necessarily be faster overall. Lots of profiling and tuning would probably be involved.

That, or your algorithm could be optimal for concurrency already - and you would see an immediate performance improvement.

Re: The fastest rm command and one of the fastest cp commands

#18

Post by preseinger »

> The key insight is that file operations in separate directories don’t (for the most part) interfere with each other, enabling parallel execution.

i'm clearly missing something here

parallel execution helps when operations are cpu bound

file operations are (almost always) io bound

and totally unclear how directories represent an "interference" boundary

bizarre

Re: The fastest rm command and one of the fastest cp commands

#20

Post by charcircuit »

preseinger wrote: post #18

> The key insight is that file operations in separate directories don’t (for the most part) interfere with each other, enabling parallel execution. i'm clearly missing something here parallel execution helps when operations are cpu bound file operations are (almost always) io bound and totally unclear how directories represent an "interference" boundary bizarre

What I would suppose is that it reduces the amount of redundant IO where a directory is edited, flushed to disk, and then later updated again. If all these updates happen in a batch there will be less IO overall.
Post reply on HN