Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RipGrep musl binaries occasionally segfault during very-large searches

github.com

Post reply on HN
Print view View original

141–150 of 216 posts

Re: RipGrep musl binaries occasionally segfault during very-large searches

#141

Post by inigyou »

shevy-java wrote: post #41
serf wrote: post #26

Earlier quoted context omitted.

energy might be the wrong metric, plenty of energy was wasted spinning up that LLM.

Yeah but the thing is that someone tries to waste time of humans. This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understa…

I think it was GZDoom where the absentee project owner suddenly came back, pushed a bunch of nonsense AI comments, so the entire actual development community just forked it again and left him to play in his sandbox? Now it's UZDoom.

Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#142

Post by inigyou »

im3w1l wrote: post #16
smallerize wrote: post #12

Earlier quoted context omitted.

Then post whatever notes were fed into the AI instead. The verbosity and self-congratulating add negative value.

What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.

Prompt an AI to figure out the bug and write a report, of course. If that is actually useful.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#143

Post by entrope »

dark-star wrote: post #73
smukherjee19 wrote: post #52

Earlier quoted context omitted.

I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…

I understand all the words in it and follow the argument, but it's still bullshit in the technical sense of being written by someone who wants to sound authoritative but doesn't ultimately care if they are right or wrong.

The write-up should have been about five paragraphs: (1) "The script at https://... generates a tree with X million files taking 20 GB total. Running ripgrep 1.2.34, compiled with musl, on my Threadripper 9876 as 'rg ...' crashes about a fourth of the time. With glibc, it does not crash." (2) "This appears to be a bug introduced by commit abcdef1234 in some_vm_call() that allows a race between [...] such that user space can see a corrupt whatever." (3) "The sequence of operations that causes this is: (show two or more kernel threads with lines interleaved to explain the bug)". (4) and (5) as needed to elaborate on those three core points.

But the actual post is long on (fluent) speculation and short on reference to actual code. That's a large part of why it's offensive slop.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#144

Post by cyberax »

smukherjee19 wrote: post #103
dark-star wrote: post #73

Earlier quoted context omitted.

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…

Thanks for the explanation. I should have looked up each term a bit more. So "backing" is the underlying physical memory the page maps to. So if I were to make sense of that piece of slop, is this what it would be? "In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 mem…

Nope. It's more complicated. The store operation does not need to trigger the fault, it can happen for other reasons.

What happens here is a bug in munmap() that happens in another thread. It corrupts the TLB, for a brief instant replacing the correct page.

So that the virtual memory page that the crashed thread tried to read its RAM, it gets replaced by the special zero-filled physical page. The kernel uses this zero page as an optimization when it needs to create a region of data filled with zeroes, so it can just map one physical page over the entire range.

It's a really low-level bug that just requires a lot of specialized knowledge just to explain. A better write-up is certainly possible but needs to have a lot of explanations to make it understandable.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#146

Post by cyberax »

entrope wrote: post #133
MBCook wrote: post #120

Earlier quoted context omitted.

It’s the kernel. Nothing a user space library can do should ever be able to call this. Just happens to be that the musl code is able to hit this and other code isn’t for some reason.

> It’s the kernel. Nothing a user space library can do should ever be able to call this. I don't follow. An application might see this kind of crash if it has a bug causing it to access a page while another thread is mapping or unmapping that page. That would be a bug in mallocng, musl or ripgrep. Or, as someone else mentioned, it could be bug in the processor's virtual memory logic that has the same effect. Why do y…

It got traced to a race condition in munmap() in the kernel. The inefficient musl allocator simply triggers it more easily.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#147

Post by inigyou »

saghm wrote: post #86
shevy-java wrote: post #39

[flagged]

The bug is in the C code that Rust is interacting with. If anything, you seem to be making an argument in favor of replacing even more of the C code with Rust so that this doesn't happen. What you seem to think is a defense of C is accidentally just providing ammunition to the "rewrite it in Rust" crusade that I'm guessing you're very much not a fan of. There are very solid arguments against rewriting everything in R…

Rust wouldn't have helped, because this is a logic bug in the kernel, that creates a memory bug in userspace. The relevant kernel code would probably be marked "unsafe" with or without the bug.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#148

Post by inigyou »

galangalalgol wrote: post #138
inigyou wrote: post #126

Earlier quoted context omitted.

I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.

A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that co…

The Linux kernel defines syscalls as its stable ABI (note: there are also non-kernel ABIs on Linux, such as Wayland) while Windows defines the DLL calls as its stable ABI.

One isn't better than the other. Linux's approach allows binaries to be fully statically linked, which is a more predictable environment for binaries, but Windows's approach composes better, as every process loads DLLs and this allows for things like graphics drivers and COM to work more reliably. As things stand on Linux you can't use the GPU in a portable statically linked app, because the kernel doesn't define the semantics of dynamic linking.

Re: RipGrep musl binaries occasionally segfault during very-large searches

#149

Post by cyberax »

sligor wrote: post #7

So, why the bug triggers only with muslc and not other libc ?

Probably because musl's allocator exposes single newly faulted pages directly to the app. Other allocators tend to pre-allocate multiple pages at a time, so that the race window is narrower.
Post reply on HN