Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RipGrep musl binaries occasionally segfault during very-large searches

github.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: RipGrep musl binaries occasionally segfault during very-large searches

#121

Post by jcupitt »

tripflag wrote: post #109
Orphis wrote: post #94

Earlier quoted context omitted.

ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case. It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve. As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curi…

For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier. There was multithreading, and…

libvips shouldn't be calling imagemagick for heif decode, it has a nice one built in. Unless you were using a very old libvips!

I've found jemalloc works best for long running libvips processes, fwiw.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#122

Post by 3eb7988a1663 »

danudey wrote: post #113
tripflag wrote: post #109

Earlier quoted context omitted.

For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier. There was multithreading, and…

Is that memory increase seen in peak usage or average/idle usage? For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.

On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#123

Post by sroussey »

yuliyp wrote: post #99
dark-star wrote: post #73

Earlier quoted context omitted.

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…

So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.

The way I read it was that it wrote to a page and then did a read on that page within 10 instructions and that was not enough time for the memory system to have gone through the process of creating the page backed with real memory and that the timing bug widened in Linux 7.0 such that more things like this exposed the bug. Or there was flapping in the TLB for some reason, or the locking wasn’t correct, or whatever else that the kernel devs will figure out was the cause.

I’m guessing it wrote to a page and read back zero page as the memory was in the mist of being allocated.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#124

Post by neerajsi »

skydhash wrote: post #89
neerajsi wrote: post #68

Earlier quoted context omitted.

The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some q…

> The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than…

I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do.

The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.

Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#125

Post by entrope »

mort96 wrote: post #91
Orphis wrote: post #40

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one. mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in…

The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.

The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time.

Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#126

Post by inigyou »

tialaramex wrote: post #100
Orphis wrote: post #71

Earlier quoted context omitted.

That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.

Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated. On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any ot…

I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#127

Post by inigyou »

Orphis wrote: post #70
vlovich123 wrote: post #59

Earlier quoted context omitted.

It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.

Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator. mallocng should not be used at all.

Ah, no, the problem is that the bug exists.

Another, separate, problem, noticed by people investigating this problem, is that ripgrep on musl uses a serializing allocator in the hot path.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#128

Post by cyberax »

smukherjee19 wrote: post #52
hyperpape wrote: post #3

The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis .

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

This is a pretty clear analysis if you are a kernel developer.

> a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function

A result of a machine instruction to store something in RAM got erased, when the same thread attempted to load it ~10 machine instructions later. The reason is that the physical RAM page backing that particular virutal page got replaced while the thread was running.

The fault happens in the fast path for anonymous (i.e. not backed by files on disk) pages, in the granular per-VMA (virtual memory area) locks.

> "A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?

Yes, it's typical kernel terminology. "A backing page" or a "backing file".

> This is... words strung together. Nothing more. I wonder how people read and make sense of this.

There's nothing wrong with this description. It's just written for kernel developers. It needs to be expanded and explained for people who are not.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#130

Post by gsu2 »

btown wrote: post #115
dosman33 wrote: post #111

Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple us…

Similarly, I've often wondered if this is the root cause of why GitHub has become fragile of late. All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects. And if https://isolveproblems.substack.com/p/how-mi…

> All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.

In theory, that's what git's packfiles are for: loose objects get packed together into a file, and if the file is well-arranged then objects that tend to get read together will also end up getting paged-in together (see also: https://www.kernel.org/pub/software/scm/git/docs/technical/p...).

GitHub in particular has spent a lot of time and effort on packing things well (e.g. https://github.blog/open-source/git/scaling-monorepo-mainten...), so I doubt that the "billions of tiny files" problem is at the root of their recent instability.

Post reply on HN