Love the idea and execution, don't love the misplaced apo'strophe's.
Oh no - where at?
lsr: ls with io_uring
121–130 of 177 posts
Re: lsr: ls with io_uring
#122Author of the project here! I have a little write up on this here: https://rockorager.dev/log/lsr-ls-but-with-io-uring
This is fantastic stuff. I'm doing a C++ project right now that I'm doing with an eye to eventual migration in whole or in part to Zig. My little `libevring` thing is pretty young and I'd be very open to replacing it with `ourio`.
What's your feeling on having C/C++ bindings in the project as a Zig migration path for such projects?
Re: lsr: ls with io_uring
#123Earlier quoted context omitted.
The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff. It didn't require every existing tool, such as ls, to deal with things like the server rebooting while listing a directory. (The original NFS protocol is stateless, so clients can survive server reboots.) What does vi do when the server hosting the file you…
> The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff I don't agree that it was a reasonable tradeoff. Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea. I don't think this is unique to NFS, it applies to any network filesystem you try to present as if it's a local o…
> Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea.
It's the only idea though. We don't know how to make reliable systems, other than by cobbling together a lot of unreliable ones and hoping the emergent behaviour is more reliable than that of the parts.
Re: lsr: ls with io_uring
#124Author of the project here! I have a little write up on this here: https://rockorager.dev/log/lsr-ls-but-with-io-uring
This is fantastic stuff. I'm doing a C++ project right now that I'm doing with an eye to eventual migration in whole or in part to Zig. My little `libevring` thing is pretty young and I'd be very open to replacing it with `ourio`. What's your feeling on having C/C++ bindings in the project as a Zig migration path for such projects?
I think exposing a C lib would be very nice. Feel free to open a discussion or issue on the Github.
Re: lsr: ls with io_uring
#125Earlier quoted context omitted.
> The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff I don't agree that it was a reasonable tradeoff. Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea. I don't think this is unique to NFS, it applies to any network filesystem you try to present as if it's a local o…
> Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea. It's the only idea though. We don't know how to make reliable systems, other than by cobbling together a lot of unreliable ones and hoping the emergent behaviour is more reliable than that of the parts.
I think "making an unreliable system emulate a reliable one = bad" is too simplistic a heuristic.
We do this all the time with things like ECC and retransmissions and packet recovery. This intrinsically is not bad at all, the question is: what abstraction does this expose to the higher layer.
With TCP the abstraction we expect is "pretty robust but has tail latencies, do not use for automotive networks or avionics" and that works out well. The right question IMO is always "what kind of tail behaviors does this expose, and are the consumers of the abstraction prepared for them".
Re: lsr: ls with io_uring
#126> I have no idea what lsd is doing. I haven’t read the source code, but from viewing it’s strace, it is calling clock_gettime around 5 times per file. Why? I don’t know. Maybe it’s doing internal timing of steps along the way?
Maybe calculating "X minutes/hours/days/weeks ago" thing for each timestamp? (access, create, modify, ...). Could just be an old artifact of another library function...
Re: lsr: ls with io_uring
#127Really interesting, the difference is real though I would just hope that some better coloring support could be added because I have "eza --icons=always -1" command set as my ls and it looks really good, whereas when I use lsr -1, yes the fundamental thing is same, the difference is in the coloring. Yes lsr also colors the output but it doesn't know as many things as eza does For example .opus will show up as a music…
“bat” is a pretty good modern “cat”
Re: lsr: ls with io_uring
#128Re: lsr: ls with io_uring
#129Earlier quoted context omitted.
The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff. It didn't require every existing tool, such as ls, to deal with things like the server rebooting while listing a directory. (The original NFS protocol is stateless, so clients can survive server reboots.) What does vi do when the server hosting the file you…
> The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff I don't agree that it was a reasonable tradeoff. Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea. I don't think this is unique to NFS, it applies to any network filesystem you try to present as if it's a local o…
I think it highly depends on your architecture and the scale you are pushing.
The other far-edge is the S3, where appending has just been possible within the last a few years as far as I can tell. Meanwhile editing a file requiring a full download/upload, not great either.
For the NFS case, I cannot say it's my favorite, but certainly easy to setup and run on your own. Obviously a rebooting server may cause certain issues during the unavailability, but the NFS server should be in highly-available. with NFSv4.1, you may use UDP as the primary transport, which allows you to swap/switch servers pretty quickly. (Given you connect to a DNS/FQDN rather than the IP address)
Another case is the plug and play, with NFS, UNIX permissions, ownership/group details, execute bit, etc are all preserved nicely...
Besides, you could always have a "cache" server locally. Similar to GDrive or OneDrive clients, constantly syncing back and forth, caching the data locally, using file-handles to determine locks. Works pretty well _at scale_ (ie. many concurrent users in the case of GDrive or OneDrive).
Re: lsr: ls with io_uring
#130Earlier quoted context omitted.
Yeah, I wrote this as a fun little experiment to learn more io_uring usage. The practical savings of using this are tiny, maybe 5 seconds over your entire life. That wasn't the point haha
I'd be curious to know if this helps on supercomputers, which are notorious for frequently hanging for a few seconds on an ls -l.
It could, but important to keep in mind that the filesystem architecture there is also very different with a parallel filesystem with disaggregated data and metadata.
When you run `ls -l` you could potentially be enumerating a directory with one file per rank, or worse, one file per particle or something. You could try making the read fast, but I also think that it makes no sense to have that many files: you can do things to reduce the number of files on disk. Also many are trying to push for distributed object stores instead of parallel filesystems... fun space.