Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fastest rm command and one of the fastest cp commands

alexsaveau.dev

Post reply on HN
Print view View original

31–40 of 83 posts

Re: The fastest rm command and one of the fastest cp commands

#31

Post by aix1 »

jeffrallen wrote: post #27
adrianmonk wrote: post #21

Earlier quoted context omitted.

The idea is to rename it so that you don't have to wait for it to be deleted. You can let the deletion run in the background once it has been moved out of the way.

And to be totally, ridiculously pedantic (this is HN after all) mv is a single atomic metadata operation in the filesystem, which will almost certainly go into the filesystem log without any wait on i/o. A big rm is going to generate enough metadata churn that the filesystem log will need to be synced to hardware at least once, and then you've got 2 or 3 orders of magnitude more time to wait (per sync).

Plus, if there's a complex directory tree, rm has to traverse it (to deal with all the metadata) while mv doesn't.

Re: The fastest rm command and one of the fastest cp commands

#32

Post by gambiting »

On windows I find it's much faster to use robocopy to mirror an empty folder into a full one than it is to delete the full folder than pretty much any other command. I regularly have to delete a 200GB folder with 500k+ files in it, and robocopy outperforms regular rm -r by a factor of two, and GUI shift+right click delete by a factor of 4 or 5.

Re: The fastest rm command and one of the fastest cp commands

#33

Post by nomercy400 »

gambiting wrote: post #32

On windows I find it's much faster to use robocopy to mirror an empty folder into a full one than it is to delete the full folder than pretty much any other command. I regularly have to delete a 200GB folder with 500k+ files in it, and robocopy outperforms regular rm -r by a factor of two, and GUI shift+right click delete by a factor of 4 or 5.

When deleting a 25k-file node_modules folder on windows, I use the built-in rmdir, as it is much faster than the GUI. I do not know how it compares to robocopy.

Re: The fastest rm command and one of the fastest cp commands

#34

Post by mherdeg »

jeffrallen wrote: post #25

I once wrote a command called trickle-rm, which was designed to be an i/o constrained rm, this the exact opposite of this article. I needed trickle-rm because after extensive analysis, I'd found that the i/o load of a diagnostic data cleanup job was interfering with the very tight latency requirements of the main app on the server. My first effort was "nice rm -rf $oldLogs". When I'm feeling a bit evil I will ask an…

Why didn't it work? Was try 2 ionice?

Re: The fastest rm command and one of the fastest cp commands

#35

Post by b112 »

mherdeg wrote: post #34
jeffrallen wrote: post #25

I once wrote a command called trickle-rm, which was designed to be an i/o constrained rm, this the exact opposite of this article. I needed trickle-rm because after extensive analysis, I'd found that the i/o load of a diagnostic data cleanup job was interfering with the very tight latency requirements of the main app on the server. My first effort was "nice rm -rf $oldLogs". When I'm feeling a bit evil I will ask an…

Why didn't it work? Was try 2 ionice?

Not the parent, but ionice only works with certain elevators. And elevator choice mattered a lot on spinning disks.

Re: The fastest rm command and one of the fastest cp commands

#36

Post by mort96 »

eru wrote: post #11
saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

I had to delete xcode because an update through the app store just didn't work. It had been in an "updating" state for over a week, I had given the machine full working days worth of continuous on-time, and eventually I tried to cancel the update which also didn't work. The only recourse was to `sudo rm -rf /Applications/Xcode.app`.

Not the reason OP is deleting Xcode, but you'd be surprised at how buggy a lot of stuff involving Xcode can be.

Re: The fastest rm command and one of the fastest cp commands

#37

Post by mort96 »

preseinger wrote: post #18

> The key insight is that file operations in separate directories don’t (for the most part) interfere with each other, enabling parallel execution. i'm clearly missing something here parallel execution helps when operations are cpu bound file operations are (almost always) io bound and totally unclear how directories represent an "interference" boundary bizarre

Parallel execution absolutely helps when operations are IO bound, if they're more or less independent. Making two network requests in parallel is twice as fast as making them sequentially, if the payload is small enough so that latency dominates and bandwidth is negligible.

The question is, how independent are IO operations in separate directories. And the article is claiming that they're fairly independent and don't block each other.

Re: The fastest rm command and one of the fastest cp commands

#38

Post by HeavyFeather »

eyelidlessness wrote: post #24
HeavyFeather wrote: post #16

Earlier quoted context omitted.

I don't think any GUI move items before a confirmation, so I don't see the link at all.

They almost universally do! They move stuff to a designated trash or recycle bin or whatever to stage for final deletion when you commit to it.

That's a very liberal way to define "confirmation", especially since modern OSes will delete trashed items on a rolling 30-day basis, without any further action.

To rephrase:

> mv dir .old-dir && rm -r .old-dir &

it's how most GUI desktops “delete” stuff, they just run the two actions separately

It's kind of a stretch anyway since the core idea of the command was the final `&`

Re: The fastest rm command and one of the fastest cp commands

#39

Post by WJW »

edgineer wrote: post #19
tedunangst wrote: post #5

If you need a large sub tree gone, mv dir .old-dir && rm -r .old-dir & works pretty well. Faster even than an optimized parallel unlinker.

Why would running mv before rm be faster? On what file systems is this true?

I think you missed the `&` at the end of the rm command, which runs it as a background process. It's not faster in total time, but it will allow the user to continue inputting new commands much sooner.

Re: The fastest rm command and one of the fastest cp commands

#40

Post by tyingq »

saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

The benchmarks page[1] for rmz shows it does pretty well on Linux, with many cases of 1.5x-4x vs "rm -rf". Perhaps something about Linux, or the filesystem type that was used to develop, test, and tune it.

[1] https://github.com/SUPERCILEX/fuc/tree/master/comparisons

Post reply on HN