Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RipGrep musl binaries occasionally segfault during very-large searches

github.com

Post reply on HN
Print view View original

171–180 of 216 posts

Re: RipGrep musl binaries occasionally segfault during very-large searches

#171

Post by entrope »

mort96 wrote: post #165
entrope wrote: post #125

Earlier quoted context omitted.

The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time. Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.

There is no ISO C definition of opendir(), that's a POSIX thing. But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator. Ripgrep reads through every byte of most files and matches it…

I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#172

Post by cyberax »

entrope wrote: post #161
cyberax wrote: post #146

Earlier quoted context omitted.

It got traced to a race condition in munmap() in the kernel. The inefficient musl allocator simply triggers it more easily.

It got arm-waved to a race condition in munmap(). Claude (or a distill of Claude) didn't identify a race, it just convinced itself that was the cause.

Andy Lutomirski seems to think that it is a bug: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...

I looked at the explanation, and it seems to be at very least plausible. The kernel version bisection also lines up. We'll see if the proposed patch fixes this.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#173

Post by mort96 »

entrope wrote: post #171
mort96 wrote: post #165

Earlier quoted context omitted.

There is no ISO C definition of opendir(), that's a POSIX thing. But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator. Ripgrep reads through every byte of most files and matches it…

I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.

And that's why musl's allocator's performance isn't a huge concern for ripgrep

Re: RipGrep musl binaries occasionally segfault during very-large searches

#174

Post by vintagedave »

ndesaulniers wrote: post #116

Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-... > I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human." Looks like that thread is from...today!

That was a tough read, but in all the verbiage I do not see it identifying the same code or area as the real genuine human on lore.kernel.

For those who can understand Claude better than me: did it anywhere actually identify the issue?

Re: RipGrep musl binaries occasionally segfault during very-large searches

#175

Post by Orphis »

mort96 wrote: post #173
entrope wrote: post #171

Earlier quoted context omitted.

I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.

And that's why musl's allocator's performance isn't a huge concern for ripgrep

That's to be properly measured. Assuming that the allocator will not cause issues there is to be proven.

The application I recently improved by changing the allocator had a similar profile (a C++ include scanner) and thread parallel I/O functions had terrible performance originally with mallocng. Adding threads almost had negative value because of the contention.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#176

Post by Orphis »

VorpalWay wrote: post #155
tialaramex wrote: post #100

Earlier quoted context omitted.

Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated. On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any ot…

> Rust doesn't get to override musl internals. Is malloc not a weak symbol in musl? I would expect it to be overridable like it is with glibc. Or does ripgrep only override Rust's global allocator?

Last time I checked (a while ago), it wasn't. You needed to compile musl and exclude a few files from the archive to then provide your own allocator.

Depending on the toolchain, I'm assuming you could use some tricks or hacks to make it work better, but the tooling I used (Bazel) did that automatically and I haven't bothered to look at the internals yet .

Re: RipGrep musl binaries occasionally segfault during very-large searches

#177

Post by gpm »

vintagedave wrote: post #174
ndesaulniers wrote: post #116

Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-... > I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human." Looks like that thread is from...today!

That was a tough read, but in all the verbiage I do not see it identifying the same code or area as the real genuine human on lore.kernel. For those who can understand Claude better than me: did it anywhere actually identify the issue?

Assuming Andy (the mailing list post) is right, no it didn't.

I know this because he said so here in this comment section: https://news.ycombinator.com/item?id=49134550

That said the AI was sort of directionality right... it identified roughly the right code.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#178

Post by inigyou »

dataflow wrote: post #169
galangalalgol wrote: post #138

Earlier quoted context omitted.

A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that co…

FYI, Windows's approach is better in some respects, like letting you avoid syscall overhead in some cases.

Notably, Linux has this too, but it's a special mechanism (vDSO) while on Windows it's just a normal function call with no difference from any other function.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#179

Post by gruntled-worker »

ndesaulniers wrote: post #116

Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-... > I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human." Looks like that thread is from...today!

Until ~2000, "cellphone user in public" equated with "smug asshole." We're at that icky rejection stage with AI.

Two years ago, that writeup would have been viewed as a generous gift of time to the community. Now we can't be bothered to read through it because we know where it came from, and that it's just worth $0.06 in tokens. Another reason (I think) is because we know what it portends.

In a few years, digging manually through bugs will probably be a last resort. We'll go from sneering to yawning at the reports, as another AI agent reads them and verifies the fix. Just like we ignore compiler-generated assembly code and people on their cellphones.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#180

Post by vlovich123 »

saghm wrote: post #151
inigyou wrote: post #147

Earlier quoted context omitted.

Rust wouldn't have helped, because this is a logic bug in the kernel, that creates a memory bug in userspace. The relevant kernel code would probably be marked "unsafe" with or without the bug.

> Rust wouldn't have helped, because this is a logic bug in the kernel, that creates a memory bug in userspace. Unless I'm misunderstanding, the bug is kernel code accessing out-of-bounds memory in an array. That would 100% not segfault in normal Rust code. > The relevant kernel code would probably be marked "unsafe" with or without the bug. I mean, sure, if you remove the safety rails that prevent you from running o…

The kernel bug I believe hasn’t been root caused quite yet although the LLM does seem to have potentially found an unrelated problem.
Post reply on HN