Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

lsr: ls with io_uring

rockorager.dev

Post reply on HN
Print view View original

171–177 of 177 posts

Re: lsr: ls with io_uring

#171

Post by arghwhat »

neuroelectron wrote: post #113
arghwhat wrote: post #19

Earlier quoted context omitted.

> Perhaps there is some issue with using io_uring this way, perhaps vulnerabilities are exposed. ... no. It's just not interesting or particularly valuable to optimize ls, and Jens probably just used it as a demo and didn't want to keep it around.

Explicit Vulnerabilities (Documented CVEs and Exploits) These are actual discovered vulnerabilities, typically assigned CVEs and often exploited in sandbox escapes or privilege escalations: 1. CVE-2021-3491 (Kernel 5.11+) Type: Privilege escalation Mechanism: Failure to check CAP_SYS_ADMIN before registering io_uring restrictions allowed unprivileged users to bypass sandboxing. Impact: Bypass of security policy mecha…

You are not providing any relevant information. The question wasn't if io_uring could have issues, but whether Jens was hiding vulnerabilities revealed by his own implementation of ls by removing it.

Which is absolutely not the case, as that is not a way any respectable developer would handle a security issue.

Re: lsr: ls with io_uring

#172

Post by ThePallas »

fpoling wrote: post #78
SillyUsername wrote: post #3

Love it. I'm trying to understand why all command line tools don't use io_uring. As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak. If I use tools with aio or io_uring I get 1005MB/s. I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.

io_uring is the asynchronous interface and that requires to use even-based architecture to use it effectively. But many command-line tools are still written is a straightforward sequential style. If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port. But without that a very significant refactoring is necessary. Besides, io_uring is not yet stable and wh…

As an example of pretending to do exactly that, this project (author: me) combines io_uring and ucontext.. https://github.com/pallas/ioucontext/

Re: lsr: ls with io_uring

#173

Post by cbsmith »

ninkendo wrote: post #102
mprovost wrote: post #75

Earlier quoted context omitted.

The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff. It didn't require every existing tool, such as ls, to deal with things like the server rebooting while listing a directory. (The original NFS protocol is stateless, so clients can survive server reboots.) What does vi do when the server hosting the file you…

> The designers of NFS chose to make a distributed system emulate a highly consistent and available system (a hard drive), which was (and is) a reasonable tradeoff I don't agree that it was a reasonable tradeoff. Making an unreliable system emulate a reliable one is the very thing I find to be a bad idea. I don't think this is unique to NFS, it applies to any network filesystem you try to present as if it's a local o…

So what if it is a fiber channel over ethernet drive?

I got bad news for you: they're always unreliable.

I've got a setup where the network filesystem is more reliable than the local filesystem. The local filesystem is running on a RAID-0 with fiber channel over ethernet and the network storage is a RAID-5 of a bunch of RAM block devices all linked over infiniband.

Your typical SSD these days is effectively a very complex distributed RAID, but it pretends to be a very simple local disk... and that's what most local filesystems are running on.

Honestly, NFS was designed to run over UDP originally. Failure was totally an option. When I was in school, the NFS filesystem was way more reliable than the local filesystems on the same computer. Yes, distributed filesystems are hard, but... NFS is comparatively straightforward for a lot of what passes for "reliable" these days.

Re: lsr: ls with io_uring

#174

Post by burntsushi »

Sesse__ wrote: post #154
burntsushi wrote: post #144

Earlier quoted context omitted.

For many workloads, ripgrep spends the vast majority of its time searching through files. But more practically, it would be a terror to implement. ripgrep is built on top of platform specific standard file system APIs. io_uring would mean a whole heap of code to work with a different syscall pattern in addition to the existing code pattern for non-Linux targets. So to even figure out whether it would be worth doing t…

I believe that io_uring does not support getdents (despite multiple patch series being proposed). So you'd get async stat(), if you need them, but nothing else.

Oh. Well, that's definitely a deal breaker then. ripgrep already avoids stat calls on most files.

Re: lsr: ls with io_uring

#175

Post by snapplebobapple »

This is cool. I really miss the eza coloring of the file permissions/ownership/size/dates. I've installed it anyway to try when eza -l is eating it because of a large number of files in the directory.

Side note: would something like this make ncdu faster? I run into large file counts = slow using that a lot more than the ls command.

Re: lsr: ls with io_uring

#176

Post by michaelcampbell »

flemhans wrote: post #169
michaelcampbell wrote: post #168

This is awesome for learning and a funzies app. Would be nice to see it moved into some mainstream distro, to be honest. That said, in over 35 years of using `ls`, never once have I bemoaned the speed of it.

Using ls in a dir with millions of files may stall a good while.

Sure; I guess scaling personal things to Google scale just isn't something I've had to deal with as a pleb software dev/user.

Re: lsr: ls with io_uring

#177

Post by seanw444 »

Retr0id wrote: post #104
seanw444 wrote: post #100

Earlier quoted context omitted.

Is there any actual focus on ATProto as a decentralized protocol? So far it seems like its only purpose is building Bluesky as a centralized service, which I have no interest in at all.

Doesn't the existence of tangled answer your question?

No, because when I go to sign up for the service, it asks for a Bluesky account name specifically. I don't care for yet another Facebook sign-in. I want something on par with email.
Post reply on HN