Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RipGrep musl binaries occasionally segfault during very-large searches

github.com

Post reply on HN
Print view View original

151–160 of 216 posts

Re: RipGrep musl binaries occasionally segfault during very-large searches

#151

Post by saghm »

inigyou wrote: post #147
saghm wrote: post #86

Earlier quoted context omitted.

The bug is in the C code that Rust is interacting with. If anything, you seem to be making an argument in favor of replacing even more of the C code with Rust so that this doesn't happen. What you seem to think is a defense of C is accidentally just providing ammunition to the "rewrite it in Rust" crusade that I'm guessing you're very much not a fan of. There are very solid arguments against rewriting everything in R…

Rust wouldn't have helped, because this is a logic bug in the kernel, that creates a memory bug in userspace. The relevant kernel code would probably be marked "unsafe" with or without the bug.

> Rust wouldn't have helped, because this is a logic bug in the kernel, that creates a memory bug in userspace.

Unless I'm misunderstanding, the bug is kernel code accessing out-of-bounds memory in an array. That would 100% not segfault in normal Rust code.

> The relevant kernel code would probably be marked "unsafe" with or without the bug.

I mean, sure, if you remove the safety rails that prevent you from running off the side of the cliff, then you will in practice likely ending up running off the cliff. If you have to explicitly do that though, it still makes it a lot more obvious that this is a risk than if you refuse to have safety rails anyway for any purpose.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#152

Post by skydhash »

neerajsi wrote: post #124
skydhash wrote: post #89

Earlier quoted context omitted.

> The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than…

I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved mo…

> Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain.

> As a system becomes mature, the bugs become harder and harder individually to track down.

What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.

Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?

Re: RipGrep musl binaries occasionally segfault during very-large searches

#153

Post by freedomben »

csense wrote: post #105
smukherjee19 wrote: post #52

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it. MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses…

Thank you for taking the time to write that out. I found it incredibly helpful, just wanted to let you know I appreciated it!

Re: RipGrep musl binaries occasionally segfault during very-large searches

#154

Post by yuliyp »

sroussey wrote: post #123
yuliyp wrote: post #99

Earlier quoted context omitted.

So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.

The way I read it was that it wrote to a page and then did a read on that page within 10 instructions and that was not enough time for the memory system to have gone through the process of creating the page backed with real memory and that the timing bug widened in Linux 7.0 such that more things like this exposed the bug. Or there was flapping in the TLB for some reason, or the locking wasn’t correct, or whatever el…

But the write should have triggered a page fault immediately in that case, and should have been blocked on the page being actually allocated before resuming.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#155

Post by VorpalWay »

tialaramex wrote: post #100
Orphis wrote: post #71

Earlier quoted context omitted.

That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.

Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated. On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any ot…

> Rust doesn't get to override musl internals.

Is malloc not a weak symbol in musl? I would expect it to be overridable like it is with glibc. Or does ripgrep only override Rust's global allocator?

Re: RipGrep musl binaries occasionally segfault during very-large searches

#156

Post by searealist »

Orphis wrote: post #40

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one. mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in…

There is a real tradeoff:

- The musl allocator is only slow with multi-threading.

- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator.

Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#157

Post by samatman »

smukherjee19 wrote: post #104
samatman wrote: post #102

Earlier quoted context omitted.

I don't care if the cat is black or white, so long as it catches mice. Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.

Unfortunately, the kernel bug would need to be understood to be patched, and the developed patch itself would need to be reviewed before being accepted, which seems... very hard, to put it mildly, using the nails-on-chalkboard LLM spew report.

Yes, the word 'if' was in that sentence.

The only people who have a need to judge the quality of the report (not aesthetically, I mean) are those few who know enough about the subsystems of the kernel it implicates to do something about it.

My point: does it lead to a bugfix? Good. No? Bad.

Do you have an answer to that question? I believe it's a bit early, no?

Re: RipGrep musl binaries occasionally segfault during very-large searches

#158

Post by kelnos »

hyperpape wrote: post #3

The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis .

Not really. It's typical rambling LLM slop-analysis. It may be a correct analysis (I couldn't stomach reading it in detail), but it's a pain in the ass to read. A human doing the same analysis would have written something 1/5th the length.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#159

Post by karel-3d »

vlovich123 wrote: post #59
Orphis wrote: post #40

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one. mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in…

It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.

It's maybe a kernel bug.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#160

Post by kelnos »

csense wrote: post #105
smukherjee19 wrote: post #52

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it. MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses…

I understood the jargon just fine. The writing is still terrible and way too verbose.
Post reply on HN