The fastest rm command and one of the fastest cp commands • Hacker News Board

Re: The fastest rm command and one of the fastest cp commands

#21

Post by adrianmonk » Sun, Mar 26, 2023, 5:32 AM UTC

edgineer wrote: post #19

tedunangst wrote: post #5

If you need a large sub tree gone, mv dir .old-dir && rm -r .old-dir & works pretty well. Faster even than an optimized parallel unlinker.

Why would running mv before rm be faster? On what file systems is this true?

The idea is to rename it so that you don't have to wait for it to be deleted. You can let the deletion run in the background once it has been moved out of the way.

Re: The fastest rm command and one of the fastest cp commands

#22

Post by guessmyname » Sun, Mar 26, 2023, 5:48 AM UTC

LeoPanthera wrote: post #15

eru wrote: post #11

Earlier quoted context omitted.

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

Not OP, but disk space maybe? It's pretty big. $ du -Ash Xcode.app 22G Xcode.app

> Not OP, but disk space maybe? It's pretty big.

> $ du -Ash Xcode.app

> 22G Xcode.app

My installation of Xcode only takes up 3.6G and is still fully operational.

As a developer of both desktop and mobile apps, I discovered some time ago that I could delete most, if not all, of the Xcode simulators that I don’t use without affecting the functionality of the IDE. The software is smart enough to download additional assets when you attempt to launch one of the simulators. I believe that, by default, Xcode comes with simulators for the latest versions of iOS, iPad OS, Watch OS, and Apple TV, sometimes including two or three versions for each platform.

I think this is the reason why your installation weights 22G.

Re: The fastest rm command and one of the fastest cp commands

#23

Post by noam_k » Sun, Mar 26, 2023, 5:53 AM UTC

tedunangst wrote: post #5

If you need a large sub tree gone, mv dir .old-dir && rm -r .old-dir & works pretty well. Faster even than an optimized parallel unlinker.

Very nice!

This reminds me of a similar trick to speed up deallocations by moving them to a new thread: https://abramov.io/rust-dropping-things-in-another-thread

Re: The fastest rm command and one of the fastest cp commands

#24

Post by eyelidlessness » Sun, Mar 26, 2023, 6:17 AM UTC

HeavyFeather wrote: post #16

eyelidlessness wrote: post #9

Earlier quoted context omitted.

Amusingly it’s basically how most GUI desktops “delete” stuff, they just put a confirmation in place of the &&

I don't think any GUI move items before a confirmation, so I don't see the link at all.

They almost universally do! They move stuff to a designated trash or recycle bin or whatever to stage for final deletion when you commit to it.

Re: The fastest rm command and one of the fastest cp commands

#25

Post by jeffrallen » Sun, Mar 26, 2023, 6:28 AM UTC

I once wrote a command called trickle-rm, which was designed to be an i/o constrained rm, this the exact opposite of this article. I needed trickle-rm because after extensive analysis, I'd found that the i/o load of a diagnostic data cleanup job was interfering with the very tight latency requirements of the main app on the server. My first effort was "nice rm -rf $oldLogs". When I'm feeling a bit evil I will ask an interview candidate why that didn't work.

Always something interesting to learn at the margins

Re: The fastest rm command and one of the fastest cp commands

#26

Post by renox » Sun, Mar 26, 2023, 6:31 AM UTC

tedunangst wrote: post #5

If you need a large sub tree gone, mv dir .old-dir && rm -r .old-dir & works pretty well. Faster even than an optimized parallel unlinker.

Yes, I use this trick at work to remove the generated build directory. IMHO this behaviour should be added to 'rm', but not by default due to the background work triggered.

Re: The fastest rm command and one of the fastest cp commands

#27

Post by jeffrallen » Sun, Mar 26, 2023, 6:34 AM UTC

adrianmonk wrote: post #21

edgineer wrote: post #19

Earlier quoted context omitted.

Why would running mv before rm be faster? On what file systems is this true?

The idea is to rename it so that you don't have to wait for it to be deleted. You can let the deletion run in the background once it has been moved out of the way.

And to be totally, ridiculously pedantic (this is HN after all) mv is a single atomic metadata operation in the filesystem, which will almost certainly go into the filesystem log without any wait on i/o. A big rm is going to generate enough metadata churn that the filesystem log will need to be synced to hardware at least once, and then you've got 2 or 3 orders of magnitude more time to wait (per sync).

Re: The fastest rm command and one of the fastest cp commands

#28

Post by Dylan16807 » Sun, Mar 26, 2023, 7:15 AM UTC

eyelidlessness wrote: post #24

HeavyFeather wrote: post #16

Earlier quoted context omitted.

I don't think any GUI move items before a confirmation, so I don't see the link at all.

They almost universally do! They move stuff to a designated trash or recycle bin or whatever to stage for final deletion when you commit to it.

Windows, if asking is enabled, will ask before moving into the recycle bin.

Once the file is in the recycle bin, it will probably be months before final deletion happens, and windows will not ask before doing so.

Re: The fastest rm command and one of the fastest cp commands

#29

Post by blowski » Sun, Mar 26, 2023, 7:24 AM UTC

eru wrote: post #11

saagarjha wrote: post #10

Hmm. I appreciate the cute name for the project but I tested this on my machine (a Mac) and the results were not very impressive. I sacrificed a few of my SSD cycles to test how this is on deleting Xcode (for those unaware, it's a 11 GB mess of several hundred thousand files of varying sizes). Here are the results: $ time rm -rf Xcode.app real 0m39.850s user 0m0.429s sys 0m29.153s $ time rmz Xcode.app real 0m36.476s…

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

Not the OP’s reason it seems, but I have to delete Xcode quite more frequently than you’d think because it gets into a weird state during updates. This seems to happen once or twice every year.

Re: The fastest rm command and one of the fastest cp commands

#30

Post by jeffreygoesto » Sun, Mar 26, 2023, 7:56 AM UTC

edgineer wrote: post #19

tedunangst wrote: post #5

If you need a large sub tree gone, mv dir .old-dir && rm -r .old-dir & works pretty well. Faster even than an optimized parallel unlinker.

Why would running mv before rm be faster? On what file systems is this true?

It's not faster, just somewhere else.