Live data from Hacker News

RipGrep musl binaries occasionally segfault during very-large searches

github.com

101–110 of 216 posts

Re: RipGrep musl binaries occasionally segfault during very-large searches

#101
post #80

Earlier quoted context omitted.

The worst thing is that I know there's something interesting in there but I'm too arsed to take the time and decode what data the author must have used to generate that text.

Exactly, it would be so much better if they shared the prompts instead of the output, at least then we'd be able to make some sense of what they were thinking about.

Unfortunately the prompt may have been something like "investigate this bug and file an issue when you think you understand what's going on". The "better just to share the prompt" heuristic falls down when the actual underlying intellectual work was also done by the machine.

(Feel free to pretend that I used some phrase other than "intellectual work" if you dislike seeing it used to describe something done by AI.)

Re: RipGrep musl binaries occasionally segfault during very-large searches

#102

The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis .

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

I don't care if the cat is black or white, so long as it catches mice.

Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#103

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…

Thanks for the explanation. I should have looked up each term a bit more.

So "backing" is the underlying physical memory the page maps to.

So if I were to make sense of that piece of slop, is this what it would be?

"In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 memory address by another thread due to race condition, resulting in a crash when the original thread tries to read that page again."

Re: RipGrep musl binaries occasionally segfault during very-large searches

#104

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

I don't care if the cat is black or white, so long as it catches mice. Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.

Unfortunately, the kernel bug would need to be understood to be patched, and the developed patch itself would need to be reviewed before being accepted, which seems... very hard, to put it mildly, using the nails-on-chalkboard LLM spew report.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#105

The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis .

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it.

MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses used by a program map to physical RAM.

Page: A 4k region of memory addresses.

Anonymous: The page is just regular old memory used by a single process for general purposes, it's not part of e.g. a memory-mapped file.

Backing: The memory corresponding to a page. The kernel can direct the MMU to map any page to any part of RAM, or delete the mapping entirely.

Zero page: The kernel keeps a single 4 KB page filled with zeros (the "zero page").

Page fault: An error that occurs when you try to write a page that's not present, or write to a read-only page. Some page faults are normal and expected by the kernel, they support features like swapping, memory-mapped files, etc. Such "expected" page faults are invisible to your program. (On the other hand, an unexpected page fault occurs when your program just straight-up accesses an address it's not supposed to, which will end your program with a SIGSEGV signal -- an all-too-familiar experience for a C programmer.)

When you allocate say 4 MB of memory, the kernel simply sets it to 1k read-only copies of the 4 KB zero page. Then when your program tries to write to a page, it faults. The kernel handles this expected fault by assigning that page an individualized memory region. If a program allocates a lot of memory, it's only assigned memory for what it actually uses, when it first uses it. This is an optimization: Every program that allocates more memory than it needs is wasteful, but the kernel can recover that waste by not backing the allocation with memory -- the program has to prove each page is needed by writing to it.

Store to a freshly-faulted anonymous page: The program stored data to its own memory, which caused an expected page fault.

> becomes invisible to that same thread's reload ~10 instructions later

The thread fairly quickly tried to read the data from the same place in memory it had just written to. It doesn't get the same data back. This is a major problem that causes the program to malfunction and crash.

> A pagemap read at the instant of the fault shows the backing is the kernel's zero page

The expected fault occurred because the program tried to write to an address that was mapped to the zero page, probably because it's the first write to freshly allocated memory. (This rules out other reasons it might be faulted, e.g. the page had been swapped to disk.)

> concurrent munmap's TLB shootdown

The page table is big and slow so the CPU caches parts of it in an area called the TLB (translation lookaside buffer). munmap is a kernel function that removes the backing for an address region. Since munmap changes the mapping, it has to inform the MMU the relevant part of the TLB is no longer valid.

> per-VMA-lock anonymous-fault fast path

This is what the kernel does when the program first writes to a zero page. Saying it's the "fast path" implies there's some (slow) special case handling in the kernel code, but the bug doesn't trigger any of the special cases, we're in the most common case, which the kernel code tries to handle as quickly as possible because it has measurable impact on performance. I would guess per-VMA-lock has to do with how the kernel synchronizes this code (a "lock" is a basic synchronization primitive, you should be familiar with it if you do any kind of multithreaded programming).

> The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction

The mechanism: What's actually happening to cause the program not to be able to load data it just stored in memory.

Localizes: The cause of a problem like this is a needle in a haystack -- it could be caused by the program, the kernel or the hardware. The analysis has narrowed down the haystack to a specific part of the kernel code.

Race in that interaction: Two parts of the kernel code, (1) The kernel code for munmap and (2) the kernel code to handle a program's first write to the read-only zero page for a fresh allocation. These two parts work fine individually but the problem occurs when they both try to change the page tables / TLB at exactly the same time. This is quite a small, specific haystack compared to "somewhere in the program, kernel or hardware" we started with.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#106
post #66
post #8

Earlier quoted context omitted.

Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM cl…

I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357 If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd…

Not enough people are going to see this comment. It's so true!

Re: RipGrep musl binaries occasionally segfault during very-large searches

#107

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

I don't care if the cat is black or white, so long as it catches mice. Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.

[dead]

Re: RipGrep musl binaries occasionally segfault during very-large searches

#108
post #16

Earlier quoted context omitted.

Then post whatever notes were fed into the AI instead. The verbosity and self-congratulating add negative value.

What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.

"it will be"

Trying to figure out what to do based on possible future scenarios has a place, but not here when we are talking about a concrete present problem.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#109
post #94

Earlier quoted context omitted.

it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.

ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case. It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve. As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curi…

For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier.

There was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case.

The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#110
post #30

Earlier quoted context omitted.

The overflow would still overflow; the use-after-free would still use after free; a musl mask race would still race. Hilarious. Apart from the computer poetry, the conclusion seems to be “it’s something in Linux 7.0 + musl 1.2.5”, although the only reproduction is still on the same physical Threadripper CPU and only sometimes when heavily exercised, so it hasn’t really ruled out a hardware issue.

The computer seems to have identified source code to blame though? Unless of course it hallucinated which there’s always a non zero chance of

I mean, we are deep into reading tea leaves at this point, but what it seems to have identified is the source file that changed such that this issue can now occur. My gut instinct is that the changed code is still valid and correct, but it exercises the hardware in a different or a more intense way.
Post reply on HN