Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RipGrep musl binaries occasionally segfault during very-large searches

github.com

Post reply on HN
Print view View original

91–100 of 216 posts

Re: RipGrep musl binaries occasionally segfault during very-large searches

#91

Post by mort96 »

Orphis wrote: post #40

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one. mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in…

The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#92

Post by tripflag »

Orphis wrote: post #70
vlovich123 wrote: post #59

Earlier quoted context omitted.

It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.

Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator. mallocng should not be used at all.

it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#93

Post by 27183 »

im3w1l wrote: post #16
smallerize wrote: post #12

Earlier quoted context omitted.

Then post whatever notes were fed into the AI instead. The verbosity and self-congratulating add negative value.

What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.

> What will you do when the prompt was "Figure out the bug and write a report for me".

The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.

If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#94

Post by Orphis »

tripflag wrote: post #92
Orphis wrote: post #70

Earlier quoted context omitted.

Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator. mallocng should not be used at all.

it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.

ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case.

It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve.

As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curious to know about the magnitudes we're talking about. From 10MB to 20MB or from 100MB to 2GB? How was the speed of the program? Was there any multithreading involved?

Re: RipGrep musl binaries occasionally segfault during very-large searches

#95

Post by zeuxcg »

Orphis wrote: post #40

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one. mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in…

If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to.

It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#96

Post by 3eb7988a1663 »

tripflag wrote: post #92
Orphis wrote: post #70

Earlier quoted context omitted.

Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator. mallocng should not be used at all.

it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.

I imagine there is no free lunch in the allocator space, but a series of trade-offs. Pick your poison on implementation which is going to be sub-optimal for some subset of use cases.

Without a hugely compelling reason to switch, going with the default is reasonable.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#97

Post by dguest »

p0nce wrote: post #37
hyperpape wrote: post #3

The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis .

Reading an AI-generated bug report is awful.

Maybe it's not going to get any better. LLMs were trained to do two things:

- Mimic human language (and logic, since that's part of what we express with language). This includes code written by humans.

- Write code to solve problems. The figure of merit here is solving the problem, not reproducing anything human.

I'm not an expert on LLMs, but it's not obvious that the human reasoning they are fitting in the first case is going to be anything like the problem solving required in the second case. It was trained to get itself out of a problem, not to do it in a way that a human would relate to.

And we're already running out of human data to train on, while synthetic data has no limit. Bug reports in the future might amount to "fix this because then I'll get a cookie".

Re: RipGrep musl binaries occasionally segfault during very-large searches

#98

Post by Banditoz »

momocowcow wrote: post #87
bentcorner wrote: post #67

Earlier quoted context omitted.

The worst thing is that I know there's something interesting in there but I'm too arsed to take the time and decode what data the author must have used to generate that text.

Get your own AI to give you the tldr :/

How does that help anything?

Re: RipGrep musl binaries occasionally segfault during very-large searches

#99

Post by yuliyp »

dark-star wrote: post #73
smukherjee19 wrote: post #52

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…

So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#100

Post by tialaramex »

Orphis wrote: post #71
masklinn wrote: post #63

Earlier quoted context omitted.

> I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). ripgrep actually sets jemalloc as global allocator when built for 64b musl: https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab...

That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.

Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated.

On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.

Post reply on HN