81–90 of 177 posts
Earlier quoted context omitted.
Well I have a directory with a couple million JSON files and ls/du take minutes. Most of the coreutils are not fast enough to actually utilize modern SSDs.
What’s the filesystem type? Ext4 suffers terrible lookup performance with large directories, while xfs absolutely flies.
Earlier quoted context omitted.
In the history of Unix its also a common way to propose tool replacements, for instance how `less` became `more` on most systems, or `vim` became the new `vi` which in its day became the new `ed`.
> instance how `less` became `more` on most systems How `more` became `less`. The name of 'more' was from paging - rather than having text scroll off the screen, it would show you one page, then ask if you wanted to see 'more' and scroll down. 'less' is a joke by the less authors. 'less is more' etc.
Earlier quoted context omitted.
> 35x less system calls = others wait less for the kernel to handle their system calls That isn't how it works. There isn't a fixed syscall budget distributed among running programs. Internally, the kernel is taking many of the same locks and resources to satisfy io_uring requests as ordinary syscall requests.
More system calls mean more overall OS overhead eg. more context switches, or as you say more contention on internal locks etc. Also, more fs-related system calls mean less available kernel threads to process these system calls. eg. XFS can paralellize mutations only up to its number of allocation groups (agcount)
Again, this just isn't true. The same "stat" operations are being performed one way or another.
> Also, more fs-related system calls mean less available kernel threads to process these system calls.
Generally speaking sync system calls are processed in the context of the calling (user) thread. They don't consume kernel threads generally. In fact the opposite is true here -- io_uring requests are serviced by an internal kernel thread pool, so to the extent this matters, io_uring requests consume more kernel threads.
Lovely, I might try doing this for some other "classic" utility! A bit off-topic too, but I'm new to Zig and curious. This here: ``` const allocator = sfb.get(); var cmd: Command = .{ .arena = allocator }; ``` means that all allocations need to be written with an allocator in mind? I.e. one has to pick an allocator per each memory allocation? Or is there a default one?
Caveat emptor, I don't write Zig but followed its development closely for awhile. A core design element of zig is that you shouldn't be stuck with one particular memory model. Zig encourages passing an allocator context around, where those allocators conform to a standardized interface. That means you could pass in different allocators with different performance characteristics at runtime. But yes, there is a default…
Nit: an allocator is not a "memory model", and I very much want the memory model to not change under my feet.
Earlier quoted context omitted.
More system calls mean more overall OS overhead eg. more context switches, or as you say more contention on internal locks etc. Also, more fs-related system calls mean less available kernel threads to process these system calls. eg. XFS can paralellize mutations only up to its number of allocation groups (agcount)
> More system calls mean more overall OS overhead [than the equivalent operations performed with io_uring] Again, this just isn't true. The same "stat" operations are being performed one way or another. > Also, more fs-related system calls mean less available kernel threads to process these system calls. Generally speaking sync system calls are processed in the context of the calling (user) thread. They don't consume…
Again, it just is true.
More fs-related operations mean less kthreads available for others. More syscalls means more OS overhead. It's that simple.
Love it. I'm trying to understand why all command line tools don't use io_uring. As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak. If I use tools with aio or io_uring I get 1005MB/s. I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.
iirc io_uring also had some pretty significant security issues early on (a couple of years ago). Those should be fixed by now, but that probably dampened adoption as well.
Lovely, I might try doing this for some other "classic" utility! A bit off-topic too, but I'm new to Zig and curious. This here: ``` const allocator = sfb.get(); var cmd: Command = .{ .arena = allocator }; ``` means that all allocations need to be written with an allocator in mind? I.e. one has to pick an allocator per each memory allocation? Or is there a default one?
Caveat emptor, I don't write Zig but followed its development closely for awhile. A core design element of zig is that you shouldn't be stuck with one particular memory model. Zig encourages passing an allocator context around, where those allocators conform to a standardized interface. That means you could pass in different allocators with different performance characteristics at runtime. But yes, there is a default…
in libraries. if youre just writing a final product it's totally fine to pick one and use it everywhere.
> std.heap.page_allocator
strongly disrecommend using this allocator as "default", it will take a trip to kernelland on each allocation.
This seems more interesting as demonstration of the amortized performance increase you'd expect from using io_uring, or as a tutorial for using it. I don't understand why I'd switch from using something like eza. If I'm listing 10,000 files the difference is between 40ms and 20ms. I absolutely would not notice that for a single invocation of the command.