Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fastest rm command and one of the fastest cp commands

alexsaveau.dev

Post reply on HN
Print view View original

41–50 of 83 posts

Re: The fastest rm command and one of the fastest cp commands

#41

Post by criddell »

gambiting wrote: post #32

On windows I find it's much faster to use robocopy to mirror an empty folder into a full one than it is to delete the full folder than pretty much any other command. I regularly have to delete a 200GB folder with 500k+ files in it, and robocopy outperforms regular rm -r by a factor of two, and GUI shift+right click delete by a factor of 4 or 5.

I’m going to have to try that. I want to throw my computer out the window when I delete a folder in File Explorer and I get the “Preparing to delete” dialog come up. If I’m truly deleting and not recycling, what is it preparing?

Re: The fastest rm command and one of the fastest cp commands

#42

Post by oefrha »

mort96 wrote: post #36
eru wrote: post #11

Earlier quoted context omitted.

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

I had to delete xcode because an update through the app store just didn't work. It had been in an "updating" state for over a week, I had given the machine full working days worth of continuous on-time, and eventually I tried to cancel the update which also didn't work. The only recourse was to `sudo rm -rf /Applications/Xcode.app`. Not the reason OP is deleting Xcode, but you'd be surprised at how buggy a lot of stu…

App Store Xcode updates always take forever for god knows why. It’s almost as if there’s some nonlinear behavior in softwareupdated and co. that’s hit by Xcode with its gazillion little files.

Therefore I update Xcode by downloading the .xip from developer.apple.com/downloads.

Re: The fastest rm command and one of the fastest cp commands

#43

Post by imtringued »

preseinger wrote: post #18

> The key insight is that file operations in separate directories don’t (for the most part) interfere with each other, enabling parallel execution. i'm clearly missing something here parallel execution helps when operations are cpu bound file operations are (almost always) io bound and totally unclear how directories represent an "interference" boundary bizarre

Synchronous file system IO throughput is inversely correlated with latency. If you have a remote network then using multiple connections at the same time allows linear performance improvements until you have maxed out the throughout of your network link or remote filesystem.

Re: The fastest rm command and one of the fastest cp commands

#44

Post by jabl »

tyingq wrote: post #40
saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

The benchmarks page[1] for rmz shows it does pretty well on Linux, with many cases of 1.5x-4x vs "rm -rf". Perhaps something about Linux, or the filesystem type that was used to develop, test, and tune it. [1] https://github.com/SUPERCILEX/fuc/tree/master/comparisons

There has been a huge amount of work done on Linux over the years to increase parallelism in the VFS and in the common filesystems like ext4, XFS and btrfs.

I would guess that macOS, being targeted at desktops and the typical workloads being run on them, hasn't received the same level of scrutiny in this particular area.

Re: The fastest rm command and one of the fastest cp commands

#45

Post by gambiting »

criddell wrote: post #41
gambiting wrote: post #32

On windows I find it's much faster to use robocopy to mirror an empty folder into a full one than it is to delete the full folder than pretty much any other command. I regularly have to delete a 200GB folder with 500k+ files in it, and robocopy outperforms regular rm -r by a factor of two, and GUI shift+right click delete by a factor of 4 or 5.

I’m going to have to try that. I want to throw my computer out the window when I delete a folder in File Explorer and I get the “Preparing to delete” dialog come up. If I’m truly deleting and not recycling, what is it preparing?

It queries the list of all files and their sizes so it can show you a progress bar. RM and robocopy will just get on with it, but they don't know how much they are about to delete(not that they need to)

Re: The fastest rm command and one of the fastest cp commands

#46

Post by AlpineG »

saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

Look at user and system times in your test. rmz seems much worse by that measure, particularly syscalls its 4x more time for 4x the number of cpus, maybe a link there. evidently it is multi threaded and as you say causes contention.

Re: The fastest rm command and one of the fastest cp commands

#47

Post by redman25 »

Just started working on ‘can’, an ‘rm’ replacement, that moves files to the trash instead instead of deleting them. It only works on macos right now but I was hoping to make it cross-platform. It’s not faster than ‘rm’ but hopefully saves accidental deletions. https://github.com/joshvoigts/can

Re: The fastest rm command and one of the fastest cp commands

#48

Post by bscphil »

HeavyFeather wrote: post #38
eyelidlessness wrote: post #24

Earlier quoted context omitted.

They almost universally do! They move stuff to a designated trash or recycle bin or whatever to stage for final deletion when you commit to it.

That's a very liberal way to define "confirmation", especially since modern OSes will delete trashed items on a rolling 30-day basis, without any further action. To rephrase: > mv dir .old-dir && rm -r .old-dir & it's how most GUI desktops “delete” stuff, they just run the two actions separately It's kind of a stretch anyway since the core idea of the command was the final `&`

> modern OSes will delete trashed items on a rolling 30-day basis, without any further action.

Hmm, the best I can find, no OS does this by default although all of them are capable of doing it.

Re: The fastest rm command and one of the fastest cp commands

#49

Post by plg94 »

redman25 wrote: post #47

Just started working on ‘can’, an ‘rm’ replacement, that moves files to the trash instead instead of deleting them. It only works on macos right now but I was hoping to make it cross-platform. It’s not faster than ‘rm’ but hopefully saves accidental deletions. https://github.com/joshvoigts/can

For Linux there's [trash-cli](https://github.com/andreafrancia/trash-cli/). Doesn't seem to work for MacOS per this issue (https://github.com/andreafrancia/trash-cli/issues/284), but it suggests to use https://hasseg.org/trash/

Re: The fastest rm command and one of the fastest cp commands

#50

Post by redman25 »

plg94 wrote: post #49
redman25 wrote: post #47

Just started working on ‘can’, an ‘rm’ replacement, that moves files to the trash instead instead of deleting them. It only works on macos right now but I was hoping to make it cross-platform. It’s not faster than ‘rm’ but hopefully saves accidental deletions. https://github.com/joshvoigts/can

For Linux there's [trash-cli]( https://github.com/andreafrancia/trash-cli/ ). Doesn't seem to work for MacOS per this issue ( https://github.com/andreafrancia/trash-cli/issues/284 ), but it suggests to use https://hasseg.org/trash/

Ya, it’s mostly a learning excuse.
Post reply on HN