Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

lsr: ls with io_uring

rockorager.dev

Post reply on HN
Print view View original

61–70 of 177 posts

Re: lsr: ls with io_uring

#61

Post by api »

Why isn’t it possible — or is it — to make libc just use uring instead of syscall?

Yes I know uring is an async interface, but it’s trivial to implement sync behavior on top of a single chain of async send-wait pairs, like doing a simple single threaded “conversational” implementation of a network protocol.

It wouldn’t make a difference in most individual cases but overall I wonder how big a global speed boost you’d get by removing a ton of syscalls?

Or am I failing to understand something about the performance nuances here?

Re: lsr: ls with io_uring

#62

Post by 0x000xca0xfe »

maplant wrote: post #52

This seems more interesting as demonstration of the amortized performance increase you'd expect from using io_uring, or as a tutorial for using it. I don't understand why I'd switch from using something like eza. If I'm listing 10,000 files the difference is between 40ms and 20ms. I absolutely would not notice that for a single invocation of the command.

Well I have a directory with a couple million JSON files and ls/du take minutes.

Most of the coreutils are not fast enough to actually utilize modern SSDs.

Re: lsr: ls with io_uring

#63

Post by hansvm »

IggleSniggle wrote: post #33
quibono wrote: post #21

Lovely, I might try doing this for some other "classic" utility! A bit off-topic too, but I'm new to Zig and curious. This here: ``` const allocator = sfb.get(); var cmd: Command = .{ .arena = allocator }; ``` means that all allocations need to be written with an allocator in mind? I.e. one has to pick an allocator per each memory allocation? Or is there a default one?

Caveat emptor, I don't write Zig but followed its development closely for awhile. A core design element of zig is that you shouldn't be stuck with one particular memory model. Zig encourages passing an allocator context around, where those allocators conform to a standardized interface. That means you could pass in different allocators with different performance characteristics at runtime. But yes, there is a default…

std.heap.smp_allocator

You should basically only use the page allocator if you're writing another allocator.

Re: lsr: ls with io_uring

#64

Post by loeg »

plq wrote: post #58
loeg wrote: post #35

Earlier quoted context omitted.

Why do you say more importantly? The time is all that matters, I think.

%70 faster = you wait less 35x less system calls = others wait less for the kernel to handle their system calls

> 35x less system calls = others wait less for the kernel to handle their system calls

That isn't how it works. There isn't a fixed syscall budget distributed among running programs. Internally, the kernel is taking many of the same locks and resources to satisfy io_uring requests as ordinary syscall requests.

Re: lsr: ls with io_uring

#65

Post by ninkendo »

api wrote: post #61

Why isn’t it possible — or is it — to make libc just use uring instead of syscall? Yes I know uring is an async interface, but it’s trivial to implement sync behavior on top of a single chain of async send-wait pairs, like doing a simple single threaded “conversational” implementation of a network protocol. It wouldn’t make a difference in most individual cases but overall I wonder how big a global speed boost you’d…

In order to make this work, libc would have to:

- Start some sort of async executor thread to service the io_uring requests/responses

- Make it so every call to "normal" syscalls causes the calling thread to sleep until the result is available (that's 1 syscall)

- When the executor thread gets a result, have it wake up the original thread (that's another syscall)

So you're basically turning 1 syscall into 2 in order to emulate the legacy syscalls.

io_uring only makes sense if you're already async. Emulating sync on top of async is nearly always a terrible idea.

Re: lsr: ls with io_uring

#66

Post by loeg »

sim7c00 wrote: post #51
pjc50 wrote: post #46

Earlier quoted context omitted.

How so?

you give process direct access to a piece of kernel memory. its a reason why there is separation. thats all.

This isn't the issue; it's relatively easy to safely share some ring buffers. The issue was/is that io_uring is rapidly growing the equivalent of ~all historical Linux syscall interfaces and sometimes comparable security measures were missed on the new interfaces. (Also, stuff like seccomp filters on syscalls are kind of meaningless for io_uring.)

Re: lsr: ls with io_uring

#67

Post by loeg »

andrepd wrote: post #45
loeg wrote: post #28

Earlier quoted context omitted.

In addition, the author might not want to sign away their rights to the FSF.

What on earth are you talking about? Why would this be the case?

Are you unfamiliar with contributing to GNU projects (ls is part of GNU corutils)?

https://www.gnu.org/prep/maintain/maintain.html#Copyright-Pa...

Re: lsr: ls with io_uring

#68

Post by loeg »

the8472 wrote: post #41

io_uring doesn't support getdents though. so the primary benefit is bulk statting (ls -l). It'd be nice if we could have a getdents in flight while processing the results of the previous one.

POSIX adopting NFS' "readdirplus" operation (getdents + stat) could negate some of the benefit towards io_uring, too.

Re: lsr: ls with io_uring

#69

Post by loeg »

api wrote: post #61

Why isn’t it possible — or is it — to make libc just use uring instead of syscall? Yes I know uring is an async interface, but it’s trivial to implement sync behavior on top of a single chain of async send-wait pairs, like doing a simple single threaded “conversational” implementation of a network protocol. It wouldn’t make a difference in most individual cases but overall I wonder how big a global speed boost you’d…

In addition to sibling's concern about syscall amplification, the async just isn't useful to the application (from a latency perspective) if you just serialize a bunch of sync requests through it.

Re: lsr: ls with io_uring

#70

Post by SkiFire13 »

tln wrote: post #50

The times seem sublinear, 10k files is less than 10x 1k files. I remember getting in to a situation during the ext2 and spinning rust days where production directories had 500k files. ls processes were slow enough to overload everything. ls -F saved me there. And filesystems got a lot better at lots of files. What filesystem was used here? It's interesting how well busybox fares, it's written for size not speed iirc?

> The times seem sublinear, 10k files is less than 10x 1k files

Two points are not enough to say it's sublinear. It might very well be some constant factor that becomes less and less important the bigger the linear factor becomes.

Or in other words 10000n+C (n+C)

Post reply on HN