Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fastest rm command and one of the fastest cp commands

alexsaveau.dev

Post reply on HN
Print view View original

71–80 of 83 posts

Re: The fastest rm command and one of the fastest cp commands

#71

Post by SUPERCILEX »

preseinger wrote: post #67
SUPERCILEX wrote: post #66

Earlier quoted context omitted.

Added a small clarification: "The intuition here is that directories are a shared resource for their direct children and must therefore serialize concurrent directory-modifying operations, causing contention. In brief, file creation or deletion cannot occur at the same time within one directory." Plus accompanying benchmark: https://alexsaveau.dev/blog/projects/performance/files/fuc/f... --- > file operations are (al…

> The NVMe protocol has extremely deep queues [1] and leaving them empty means leaving performance on the table. I think you'll find it surprisingly difficult to saturate the PCIe bus with just one core: PCIe 7 will support 512GB/s is disk IO bottlenecked by NVMe/PCIe limits, or by disk iops limits? > Unix FSs tend to store file_name to inode mappings per directory, so creating a new mapping in one directory doesn't…

> is disk IO bottlenecked by NVMe/PCIe limits, or by disk iops limits?

Note that I'm out of my depth here, so this is all speculation. Until we hit hardware limitations (which will be PCIe 6 if I had to guess), I'm pretty sure those are the same thing. One read/write iop = 4KiB. If your PCIe bandwidth is limited to N GB/s, then there are only so many iops you can physically send/receive to/from the SSD, regardless of how many iops the SSD could be capable of processing. So currently we're bottlenecked by PCIe, but I doubt that will continue to be the case.

> again, you're handwaving on what "interfere" means

It depends on how the file system is implemented, but my guess would be that a lock on the inode or block cache entry is acquired.

> do "inode mappings" represent resources with no shared resource constraints?

Those are the contents of the directory. The problem is not reading them, but changing them.

Re: The fastest rm command and one of the fastest cp commands

#72

Post by SUPERCILEX »

preseinger wrote: post #70
SUPERCILEX wrote: post #69

Earlier quoted context omitted.

A directory is a file like anything else that contains a map of names to inodes. If you're trying to add or remove mappings (create or delete files), then clearly some synchronization must occur or the contents of the file will contain garbage. In theory you could get away with a very small critical section that says "lock bytes N through M" of the file, but then how do you deal with disk block alignment (i.e. two pa…

1. your benchmarks exercise a very narrow set of use cases 2. the overhead of modifying the dirent is statistically zero compared to the costs related to manipulating the files on disk

No. Run the benchmark on a tmpfs:

$ hyperfine --warmup 3 -N "./test /dev/shm 8 zip" "./test /dev/shm 8 chain" Benchmark 1: ./test /dev/shm 8 zip Time (mean ± σ): 118.5 ms ± 11.6 ms [User: 92.9 ms, System: 726.6 ms] Range (min … max): 103.6 ms … 143.4 ms 23 runs

Benchmark 2: ./test /dev/shm 8 chain Time (mean ± σ): 235.7 ms ± 11.0 ms [User: 116.4 ms, System: 1537.7 ms] Range (min … max): 220.1 ms … 258.3 ms 13 runs

Summary './test /dev/shm 8 zip' ran 1.99 ± 0.22 times faster than './test /dev/shm 8 chain'

Re: The fastest rm command and one of the fastest cp commands

#73

Post by rektide »

jlokier wrote: post #58
rektide wrote: post #54

Earlier quoted context omitted.

Historically async has had a penalty because it means doing usrerland code, then sys calling the kernel, then going back to userland, then latter somehow picking up the event, which is if nothing else just more sys calls. Which have overhead, as the processor has to context switch to do so. The whole point of io_uring is to drastically decrease the number of sys calls, creating channels where more requests can be fil…

> The whole point of io_uring is to drastically decrease the number of sys calls, creating channels where more requests can be filed with lower than traditional cost of a readFile syscall for example, and where completion can also be lower overhead delivery of events. In theory. I did some work on high performance filesystem I/O on Linux about a year ago, doing intensive random-access to fast SSDs, and found io_uring…

reply from jamie lokier will make my day any day.

not to fanboy out TOO much, but your posting on HyBi was & is greatly influential to me. see, https://github.com/rektide/pipe-layer#essence

(sorely neglected project to me but also still very near & dear, still a core principle & value in my pantheon of beliefs)]

no particular comment on io_uring. thankfully jens keeps making it better. the numbers he posts for his synthetics keep seeming impossibly good. but i fully am ready to believe the real situation is more complicated.

i do wish we'd see some uptake from the usual suspects. both Deno and Node have delayed/deferred work on these topics. but supposedly slowly happening in node. https://github.com/libuv/libuv/issues/1947 https://github.com/denoland/deno/issues/16232

Re: The fastest rm command and one of the fastest cp commands

#74

Post by saagarjha »

LeoPanthera wrote: post #15
eru wrote: post #11

Earlier quoted context omitted.

Slight tangent: why do you need to delete Xcode so often? (I don't really develop on Mac, mostly on Linux.)

Not OP, but disk space maybe? It's pretty big. $ du -Ash Xcode.app 22G Xcode.app

Xcode is compressed on disk by default; you should generally use du -sh when measuring its size.

Re: The fastest rm command and one of the fastest cp commands

#75

Post by saagarjha »

guessmyname wrote: post #22
LeoPanthera wrote: post #15

Earlier quoted context omitted.

Not OP, but disk space maybe? It's pretty big. $ du -Ash Xcode.app 22G Xcode.app

> Not OP, but disk space maybe? It's pretty big. > $ du -Ash Xcode.app > 22G Xcode.app My installation of Xcode only takes up 3.6G and is still fully operational. As a developer of both desktop and mobile apps, I discovered some time ago that I could delete most, if not all, of the Xcode simulators that I don’t use without affecting the functionality of the IDE. The software is smart enough to download additional ass…

3.6 GB is pretty small. What did you remove from your installation?

Re: The fastest rm command and one of the fastest cp commands

#76

Post by saagarjha »

oefrha wrote: post #42
mort96 wrote: post #36

Earlier quoted context omitted.

I had to delete xcode because an update through the app store just didn't work. It had been in an "updating" state for over a week, I had given the machine full working days worth of continuous on-time, and eventually I tried to cancel the update which also didn't work. The only recourse was to `sudo rm -rf /Applications/Xcode.app`. Not the reason OP is deleting Xcode, but you'd be surprised at how buggy a lot of stu…

App Store Xcode updates always take forever for god knows why. It’s almost as if there’s some nonlinear behavior in softwareupdated and co. that’s hit by Xcode with its gazillion little files. Therefore I update Xcode by downloading the .xip from developer.apple.com/downloads.

I haven't looked into this closely but I think it first tries to validate the bundle and then decompresses to disk very slowly

Re: The fastest rm command and one of the fastest cp commands

#77

Post by aeonik »

jeffrallen wrote: post #51
mherdeg wrote: post #34

Earlier quoted context omitted.

Why didn't it work? Was try 2 ionice?

Nice reduces your scheduling priority for access to the CPU, which means it might take your niced rm longer to do the processing to submit a bunch of i/o to the kernel. But once those i/o ops are in the kernel, they are competing directly on an equal playing field with the latency critical i/o ops of the real app, causing the same degradation of latency to the end users. ionice was not available on my platform, thus…

Run rm in a bash loop for each file and sync after each iteration?

I'm surprised that rm's io would cause that much of an issue. I thought it only removed entries from the partition table.

Re: The fastest rm command and one of the fastest cp commands

#78

Post by preseinger »

SUPERCILEX wrote: post #71
preseinger wrote: post #67

Earlier quoted context omitted.

> The NVMe protocol has extremely deep queues [1] and leaving them empty means leaving performance on the table. I think you'll find it surprisingly difficult to saturate the PCIe bus with just one core: PCIe 7 will support 512GB/s is disk IO bottlenecked by NVMe/PCIe limits, or by disk iops limits? > Unix FSs tend to store file_name to inode mappings per directory, so creating a new mapping in one directory doesn't…

> is disk IO bottlenecked by NVMe/PCIe limits, or by disk iops limits? Note that I'm out of my depth here, so this is all speculation. Until we hit hardware limitations (which will be PCIe 6 if I had to guess), I'm pretty sure those are the same thing. One read/write iop = 4KiB. If your PCIe bandwidth is limited to N GB/s, then there are only so many iops you can physically send/receive to/from the SSD, regardless of…

when you interact with a file or directory, the mapping of that logical thing to disk inodes (which are the physical 4kb whatevers you speak of) is managed for you by the fs cache

that cache coalesces and serializes access to disk, it does all of this "locking" you're referring to, and it's very smart

it seems like you're writing code assuming this intermediating layer does not exist?

Re: The fastest rm command and one of the fastest cp commands

#79

Post by preseinger »

SUPERCILEX wrote: post #72
preseinger wrote: post #70

Earlier quoted context omitted.

1. your benchmarks exercise a very narrow set of use cases 2. the overhead of modifying the dirent is statistically zero compared to the costs related to manipulating the files on disk

No. Run the benchmark on a tmpfs: $ hyperfine --warmup 3 -N "./test /dev/shm 8 zip" "./test /dev/shm 8 chain" Benchmark 1: ./test /dev/shm 8 zip Time (mean ± σ): 118.5 ms ± 11.6 ms [User: 92.9 ms, System: 726.6 ms] Range (min … max): 103.6 ms … 143.4 ms 23 runs Benchmark 2: ./test /dev/shm 8 chain Time (mean ± σ): 235.7 ms ± 11.0 ms [User: 116.4 ms, System: 1537.7 ms] Range (min … max): 220.1 ms … 258.3 ms 13 runs Su…

do you know what a tmpfs is? or what these programs are actually exercising?

i mean ignore me if you want, no skin off my back

but you're not benchmarking what you think you're benchmarking

Re: The fastest rm command and one of the fastest cp commands

#80

Post by preseinger »

SUPERCILEX wrote: post #69
preseinger wrote: post #68

Earlier quoted context omitted.

> The intuition here is that directories are a shared resource for their direct children and must therefore serialize concurrent directory-modifying operations, causing contention. why do you think this is true? i've never heard of anything like it directories are inodes on a file system, they are in no way "shared resources for their direct children", and there is no concept of a "directory-modifying operation" whic…

A directory is a file like anything else that contains a map of names to inodes. If you're trying to add or remove mappings (create or delete files), then clearly some synchronization must occur or the contents of the file will contain garbage. In theory you could get away with a very small critical section that says "lock bytes N through M" of the file, but then how do you deal with disk block alignment (i.e. two pa…

> A directory is a file like anything else that contains a map of names to inodes. If you're trying to add or remove mappings (create or delete files), then clearly some synchronization must occur or the contents of the file will contain garbage.

this synchronization is handled for you by the fs, specifically the fs cache

inode alignment and errors are managed by this intermediating layer

your benchmarks are not demonstrating what you think they are demonstrating

Post reply on HN