Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

lsr: ls with io_uring

rockorager.dev

Post reply on HN
Print view View original

101–110 of 177 posts

Re: lsr: ls with io_uring

#101

Post by maplant »

rockorager wrote: post #90
maplant wrote: post #52

This seems more interesting as demonstration of the amortized performance increase you'd expect from using io_uring, or as a tutorial for using it. I don't understand why I'd switch from using something like eza. If I'm listing 10,000 files the difference is between 40ms and 20ms. I absolutely would not notice that for a single invocation of the command.

Yeah, I wrote this as a fun little experiment to learn more io_uring usage. The practical savings of using this are tiny, maybe 5 seconds over your entire life. That wasn't the point haha

It's a very cool experiment. Just wanted to perhaps steer the conversation towards those things rather than whether or not this was a good ls replacement because like you say that feels like it was missing the point

Re: lsr: ls with io_uring

#102

Post by ninkendo »

mprovost wrote: post #75
ninkendo wrote: post #9

I wonder how it performs against an NFS server with lots of files, especially one over a kinda-crappy connection. Putting an unreliable network service behind blocking POSIX syscalls is one of the main reasons NFS is a terrible design choice (as can be seen by anyone who's tried to ctrl+c any app that's reading from a broken NFS folder), but I wonder if io_uring mitigates the bad parts somewhat.

The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff. It didn't require every existing tool, such as ls, to deal with things like the server rebooting while listing a directory. (The original NFS protocol is stateless, so clients can survive server reboots.) What does vi do when the server hosting the file you…

> The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff

I don't agree that it was a reasonable tradeoff. Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea. I don't think this is unique to NFS, it applies to any network filesystem you try to present as if it's a local one.

> What does vi do when the server hosting the file you're editing stop responding? None of these tools have that kind of error handling.

That's exactly why I don't think it's a good idea to just pretend a network connection is actually a local disk. Because tools aren't set up to handle issues with it being down.

Contrast it with approaches where the client is aware of the network connection (like HTTP/GRPC/etc)... the client can decide for itself how long it should retry failed requests, whether it should bubble up failures to the caller, or work "offline" until it gets an opportunity to resync, etc. With NFS the syscall just hangs forever by default.

Distributed systems are hard, and NFS (and other similar network filesystems) just pretend it isn't hard at all, which is great until something goes wrong, and then the abstraction leaks.

(Also I didn't say io_uring solves this, but I'm curious as to whether its performance would be any better than blocking calls.)

Re: lsr: ls with io_uring

#103

Post by rockorager »

tavianator wrote: post #99
rockorager wrote: post #83

Author of the project here! I have a little write up on this here: https://rockorager.dev/log/lsr-ls-but-with-io-uring

My bfs project also uses io_uring: https://github.com/tavianator/bfs/blob/main/src/ioq.c I'm curious how lsr compares to bfs -ls for example. bfs only uses io_uring when multiple threads are enabled, but maybe it's worth using it even for bfs -j1

Oh that's cool. `find` is another tool I thought could benefit from io_uring like `ls`. I think it's definitely worth enabling io_uring for single threaded applications for the batching benefit. The kernel will still spin up a thread pool to get the work done concurrently, but you don't have to manage that in your codebase.

Re: lsr: ls with io_uring

#104

Post by Retr0id »

seanw444 wrote: post #100
nikodunk wrote: post #79

Earlier quoted context omitted.

Same! Just signed up and will be following tangled and this repo. I like how tangled is built on atproto (bluesky).

Is there any actual focus on ATProto as a decentralized protocol? So far it seems like its only purpose is building Bluesky as a centralized service, which I have no interest in at all.

Doesn't the existence of tangled answer your question?

Re: lsr: ls with io_uring

#105

Post by JdeBP »

yencabulator wrote: post #82
nailer wrote: post #53

Earlier quoted context omitted.

> instance how `less` became `more` on most systems How `more` became `less`. The name of 'more' was from paging - rather than having text scroll off the screen, it would show you one page, then ask if you wanted to see 'more' and scroll down. 'less' is a joke by the less authors. 'less is more' etc.

For a while there was a less competitor named most.

It hasn't gone away.

* https://freshports.org/sysutils/most/

* https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/misc/most/i...

* https://packages.debian.org/sid/most

One can even get pg still, with Ilumos-based systems; even though that was actually taken out of the SUS years ago. This goes to show that what's standard is not the same as what exists, of course.

* https://illumos.org/man/1/pg

* https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...

Re: lsr: ls with io_uring

#107

Post by tavianator »

rockorager wrote: post #103
tavianator wrote: post #99

Earlier quoted context omitted.

My bfs project also uses io_uring: https://github.com/tavianator/bfs/blob/main/src/ioq.c I'm curious how lsr compares to bfs -ls for example. bfs only uses io_uring when multiple threads are enabled, but maybe it's worth using it even for bfs -j1

Oh that's cool. `find` is another tool I thought could benefit from io_uring like `ls`. I think it's definitely worth enabling io_uring for single threaded applications for the batching benefit. The kernel will still spin up a thread pool to get the work done concurrently, but you don't have to manage that in your codebase.

I did try it a while ago and it wasn't profitable, but that was before I added stat() support. Batching those is probably good

Re: lsr: ls with io_uring

#108

Post by mshockwave »

rockorager wrote: post #103
tavianator wrote: post #99

Earlier quoted context omitted.

My bfs project also uses io_uring: https://github.com/tavianator/bfs/blob/main/src/ioq.c I'm curious how lsr compares to bfs -ls for example. bfs only uses io_uring when multiple threads are enabled, but maybe it's worth using it even for bfs -j1

Oh that's cool. `find` is another tool I thought could benefit from io_uring like `ls`. I think it's definitely worth enabling io_uring for single threaded applications for the batching benefit. The kernel will still spin up a thread pool to get the work done concurrently, but you don't have to manage that in your codebase.

and grep / ripgrep. Or did ripgrep migrate to using io_uring already?

Re: lsr: ls with io_uring

#109

Post by izabera »

ReDress wrote: post #39

I've been playing around with io_uring for a while. Still, I am yet to come across a some tests that simulate typical real life application workload. I heard of fio but are yet to check how exactly it works and whether it might be possible to simulate real life application workload with it.

what a "real life application workload" looks like is entirely dependent on your use case, but fio is very widely used in the storage industry

it's a good first approximation to test the cartesian product of

- sequential/random

- reads/writes

- in arbitrary sizes

- with arbitrarily many workers

- with many different backends to perform such i/o including io_uring

and its reporting is solid and thorough

implementing the same for your specific workload is often not trivial at all

Re: lsr: ls with io_uring

#110

Post by burntsushi »

mshockwave wrote: post #108
rockorager wrote: post #103

Earlier quoted context omitted.

Oh that's cool. `find` is another tool I thought could benefit from io_uring like `ls`. I think it's definitely worth enabling io_uring for single threaded applications for the batching benefit. The kernel will still spin up a thread pool to get the work done concurrently, but you don't have to manage that in your codebase.

and grep / ripgrep. Or did ripgrep migrate to using io_uring already?

No, ripgrep doesn't use io_uring. Idk if it ever will.
Post reply on HN