Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fastest rm command and one of the fastest cp commands

alexsaveau.dev

Post reply on HN
Print view View original

51–60 of 83 posts

Re: The fastest rm command and one of the fastest cp commands

#51

Post by jeffrallen »

mherdeg wrote: post #34
jeffrallen wrote: post #25

I once wrote a command called trickle-rm, which was designed to be an i/o constrained rm, this the exact opposite of this article. I needed trickle-rm because after extensive analysis, I'd found that the i/o load of a diagnostic data cleanup job was interfering with the very tight latency requirements of the main app on the server. My first effort was "nice rm -rf $oldLogs". When I'm feeling a bit evil I will ask an…

Why didn't it work? Was try 2 ionice?

Nice reduces your scheduling priority for access to the CPU, which means it might take your niced rm longer to do the processing to submit a bunch of i/o to the kernel. But once those i/o ops are in the kernel, they are competing directly on an equal playing field with the latency critical i/o ops of the real app, causing the same degradation of latency to the end users. ionice was not available on my platform, thus the invention of trickle-rm.

Now the second interview question: how did trickle-rm work? How can you simply "reflect" the back pressure of the "x rm's per second" constraint back onto the "walk the directory tree to find the rm's to do" so that the tree walk generates a trickle of i/o operations?

Re: The fastest rm command and one of the fastest cp commands

#52

Post by vlovich123 »

zamalek wrote: post #17
rektide wrote: post #3

I cant wait to see the future where we have uring (io_uring) based tools that all work async, getting hopefully embarassingly parallel. Heck yes copying & deleting 2x as fast. Actually I'm on btrfs so reflink copy is like instant I think? I should test that better.

async nearly always trades concurrency for latency, so it wouldn't necessarily be faster overall. Lots of profiling and tuning would probably be involved. That, or your algorithm could be optimal for concurrency already - and you would see an immediate performance improvement.

What? How does async (and io_uring specifically) give up concurrency? They’re orthogonal things.

Re: The fastest rm command and one of the fastest cp commands

#53

Post by zamalek »

vlovich123 wrote: post #52
zamalek wrote: post #17

Earlier quoted context omitted.

async nearly always trades concurrency for latency, so it wouldn't necessarily be faster overall. Lots of profiling and tuning would probably be involved. That, or your algorithm could be optimal for concurrency already - and you would see an immediate performance improvement.

What? How does async (and io_uring specifically) give up concurrency? They’re orthogonal things.

You interpreted it the wrong way. You gain concurrency, you gain latency.

With blocking I/O and parallelism you have a thread ready to go when the operation is complete. You have N threads for N iops. With concurrency you have to dequeue completed work, and then delegate that work to (usually) fewer than N threads. Dequeuing completed iops takes time (it's an extra syscall), and there may not be a thread ready hand the completed iop. More latency.

Running 1000s of threads isn't realistic because your OS would typically grind to a halt, so concurrency is unavoidable. It does have a cost, though.

Re: The fastest rm command and one of the fastest cp commands

#54

Post by rektide »

vlovich123 wrote: post #52
zamalek wrote: post #17

Earlier quoted context omitted.

async nearly always trades concurrency for latency, so it wouldn't necessarily be faster overall. Lots of profiling and tuning would probably be involved. That, or your algorithm could be optimal for concurrency already - and you would see an immediate performance improvement.

What? How does async (and io_uring specifically) give up concurrency? They’re orthogonal things.

Historically async has had a penalty because it means doing usrerland code, then sys calling the kernel, then going back to userland, then latter somehow picking up the event, which is if nothing else just more sys calls. Which have overhead, as the processor has to context switch to do so.

The whole point of io_uring is to drastically decrease the number of sys calls, creating channels where more requests can be filed with lower than traditional cost of a readFile syscall for example, and where completion can also be lower overhead delivery of events.

So historically I kind of would have agreed with the parent. Today, we don't really know! Hence my excitement.

Re: The fastest rm command and one of the fastest cp commands

#55

Post by shaggie76 »

gambiting wrote: post #32

On windows I find it's much faster to use robocopy to mirror an empty folder into a full one than it is to delete the full folder than pretty much any other command. I regularly have to delete a 200GB folder with 500k+ files in it, and robocopy outperforms regular rm -r by a factor of two, and GUI shift+right click delete by a factor of 4 or 5.

That's a neat trick; I was in a similar situation recently and wrote a tool that's faster for me though; for a million files ROBOCOPY was 257 seconds, but https://github.com/shaggie76/FastDelete did it in 34 seconds on a hex-core laptop.

Re: The fastest rm command and one of the fastest cp commands

#56

Post by gambiting »

shaggie76 wrote: post #55
gambiting wrote: post #32

On windows I find it's much faster to use robocopy to mirror an empty folder into a full one than it is to delete the full folder than pretty much any other command. I regularly have to delete a 200GB folder with 500k+ files in it, and robocopy outperforms regular rm -r by a factor of two, and GUI shift+right click delete by a factor of 4 or 5.

That's a neat trick; I was in a similar situation recently and wrote a tool that's faster for me though; for a million files ROBOCOPY was 257 seconds, but https://github.com/shaggie76/FastDelete did it in 34 seconds on a hex-core laptop.

Oh that's super cool! Will need to try it.

Re: The fastest rm command and one of the fastest cp commands

#57

Post by ElectricalUnion »

Dylan16807 wrote: post #28
eyelidlessness wrote: post #24

Earlier quoted context omitted.

They almost universally do! They move stuff to a designated trash or recycle bin or whatever to stage for final deletion when you commit to it.

Windows, if asking is enabled, will ask before moving into the recycle bin. Once the file is in the recycle bin, it will probably be months before final deletion happens, and windows will not ask before doing so.

Unless you happen to start to run low on disk space; at that point, Storage Sense will kick in and start complaining about it - if you enable it without changing standard settings it deletes "trash bin" files every 30d.

Re: The fastest rm command and one of the fastest cp commands

#58

Post by jlokier »

rektide wrote: post #54
vlovich123 wrote: post #52

Earlier quoted context omitted.

What? How does async (and io_uring specifically) give up concurrency? They’re orthogonal things.

Historically async has had a penalty because it means doing usrerland code, then sys calling the kernel, then going back to userland, then latter somehow picking up the event, which is if nothing else just more sys calls. Which have overhead, as the processor has to context switch to do so. The whole point of io_uring is to drastically decrease the number of sys calls, creating channels where more requests can be fil…

> The whole point of io_uring is to drastically decrease the number of sys calls, creating channels where more requests can be filed with lower than traditional cost of a readFile syscall for example, and where completion can also be lower overhead delivery of events.

In theory. I did some work on high performance filesystem I/O on Linux about a year ago, doing intensive random-access to fast SSDs, and found io_uring to be slightly slower than a well-tuned thread pool with an appropriate queue depth.

That was a little surprising as the thread pool has to do system calls for each I/O operation and io_uring does not. Perhaps it is faster with newer kernels or other access patterns.

io_uring is better able to adapt autonatically to different numbers of cores, device queue depth and amount of filesystem data cache residency. That comes from it having access to kernel state which is not made available to userspace on Linux, to guide thread offloading decisions, rather than from the ringbuffer communication.

Re: The fastest rm command and one of the fastest cp commands

#59

Post by eru »

oefrha wrote: post #14
eru wrote: post #11

Earlier quoted context omitted.

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

They develop https://github.com/saagarjha/unxip , “a fast Xcode unarchiver”. Very few people should routinely delete Xcode.

Thanks!

Re: The fastest rm command and one of the fastest cp commands

#60

Post by vlovich123 »

zamalek wrote: post #53
vlovich123 wrote: post #52

Earlier quoted context omitted.

What? How does async (and io_uring specifically) give up concurrency? They’re orthogonal things.

You interpreted it the wrong way. You gain concurrency, you gain latency. With blocking I/O and parallelism you have a thread ready to go when the operation is complete. You have N threads for N iops. With concurrency you have to dequeue completed work, and then delegate that work to (usually) fewer than N threads. Dequeuing completed iops takes time (it's an extra syscall), and there may not be a thread ready hand t…

I’m not aware of any latency impact from io_uring. If anything, it has lower latency because you can pipeline I/O from a single CPU which you can’t do from typically thread-based parallelism. Additionally, the processing of the ring buffer could happen on a background kernel thread (in theory not sure if it happens today) which then avoids context switching away from your thread and screwing with cache performance.
Post reply on HN