Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

lsr: ls with io_uring

rockorager.dev

Post reply on HN
Print view View original

31–40 of 177 posts

Re: lsr: ls with io_uring

#31

Post by s1mplicissimus »

mschuster91 wrote: post #25
danbruc wrote: post #22

Why does this require inventing lsr as an alternative to ls instead of making ls use io_uring? It seems pretty annoying to have to install replacements for the most basic command line tools. And especially in this case, where you do not even do it for additional features, just for getting the exact same thing done a bit faster.

> Why does this require inventing lsr as an alternative to ls instead of making ls use io_uring? Good luck getting that upstreamed and accepted. The more foundational the tools (and GNU coreutils definitely is foundational), the more difficult that process will be. Releasing a standalone utility makes iteration much faster, partially because one is not bound to the release cycles of distributions.

> Releasing a standalone utility makes iteration much faster, partially because one is not bound to the release cycles of distributions.

which certainly is a valid way or prioritizing. similarly, distros/users may prioritize stability, which means the theoretical improvement would now be stuck in not-used-land. the value of software appears when it's run, not when it's written

Re: lsr: ls with io_uring

#32

Post by elcapitan »

SillyUsername wrote: post #3

Love it. I'm trying to understand why all command line tools don't use io_uring. As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak. If I use tools with aio or io_uring I get 1005MB/s. I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.

iirc io_uring also had some pretty significant security issues early on (a couple of years ago). Those should be fixed by now, but that probably dampened adoption as well.

Re: lsr: ls with io_uring

#33

Post by IggleSniggle »

quibono wrote: post #21

Lovely, I might try doing this for some other "classic" utility! A bit off-topic too, but I'm new to Zig and curious. This here: ``` const allocator = sfb.get(); var cmd: Command = .{ .arena = allocator }; ``` means that all allocations need to be written with an allocator in mind? I.e. one has to pick an allocator per each memory allocation? Or is there a default one?

Caveat emptor, I don't write Zig but followed its development closely for awhile. A core design element of zig is that you shouldn't be stuck with one particular memory model. Zig encourages passing an allocator context around, where those allocators conform to a standardized interface. That means you could pass in different allocators with different performance characteristics at runtime.

But yes, there is a default allocator, std.heap.page_allocator

Re: lsr: ls with io_uring

#34

Post by tln »

SillyUsername wrote: post #3

Love it. I'm trying to understand why all command line tools don't use io_uring. As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak. If I use tools with aio or io_uring I get 1005MB/s. I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.

Thats a great speed boost. What tools are these?

Re: lsr: ls with io_uring

#35

Post by loeg »

ta988 wrote: post #7
fermuch wrote: post #4

The link isn't working for me. For those who were able to see it: does it improve anything by using that instead of what ls does now??

70% faster, but more importantly 35x times less syscalls.

Why do you say more importantly? The time is all that matters, I think.

Re: lsr: ls with io_uring

#36

Post by WorldMaker »

mschuster91 wrote: post #25
danbruc wrote: post #22

Why does this require inventing lsr as an alternative to ls instead of making ls use io_uring? It seems pretty annoying to have to install replacements for the most basic command line tools. And especially in this case, where you do not even do it for additional features, just for getting the exact same thing done a bit faster.

> Why does this require inventing lsr as an alternative to ls instead of making ls use io_uring? Good luck getting that upstreamed and accepted. The more foundational the tools (and GNU coreutils definitely is foundational), the more difficult that process will be. Releasing a standalone utility makes iteration much faster, partially because one is not bound to the release cycles of distributions.

In the history of Unix its also a common way to propose tool replacements, for instance how `less` became `more` on most systems, or `vim` became the new `vi` which in its day became the new `ed`.

Re: lsr: ls with io_uring

#37

Post by johnisgood »

Imustaskforhelp wrote: post #10

Really interesting, the difference is real though I would just hope that some better coloring support could be added because I have "eza --icons=always -1" command set as my ls and it looks really good, whereas when I use lsr -1, yes the fundamental thing is same, the difference is in the coloring. Yes lsr also colors the output but it doesn't know as many things as eza does For example .opus will show up as a music…

As for coloring support, I think the best way would be to implement LS_COLORS / dircolors. My GNU ls looks nice.

Re: lsr: ls with io_uring

#39

Post by ReDress »

I've been playing around with io_uring for a while.

Still, I am yet to come across a some tests that simulate typical real life application workload.

I heard of fio but are yet to check how exactly it works and whether it might be possible to simulate real life application workload with it.

Re: lsr: ls with io_uring

#40

Post by neuroelectron »

arghwhat wrote: post #19
neuroelectron wrote: post #14

There used to be lsring by Jens Axboe (author of io_uring), but it no longer exists. This is more extreme than abandoning the project. Perhaps there is some issue with using io_uring this way, perhaps vulnerabilities are exposed.

> Perhaps there is some issue with using io_uring this way, perhaps vulnerabilities are exposed. ... no. It's just not interesting or particularly valuable to optimize ls, and Jens probably just used it as a demo and didn't want to keep it around.

I'm sure there are uses in Bash scripts that could benefit from it but most people would use it directly in a compiled program, I suppose, if the performance was a reoccurring need.
Post reply on HN