Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.
RipGrep musl binaries occasionally segfault during very-large searches
111–120 of 216 posts
Re: RipGrep musl binaries occasionally segfault during very-large searches
#112Earlier quoted context omitted.
Exactly, it would be so much better if they shared the prompts instead of the output, at least then we'd be able to make some sense of what they were thinking about.
Unfortunately the prompt may have been something like "investigate this bug and file an issue when you think you understand what's going on". The "better just to share the prompt" heuristic falls down when the actual underlying intellectual work was also done by the machine. (Feel free to pretend that I used some phrase other than "intellectual work" if you dislike seeing it used to describe something done by AI.)
Yeah in that case just closing it and moving on would be better IMO. A sufficiently large fraction of the time the issues with vibeslopped code and/or analysis are bad enough once you start digging into it that it's not worth spending time on. My heuristic would be to simply ignore contributions derived from lazy prompts. I don't think there are very many healthy babies in that bath water.
So having the prompt available would be useful even in the degenerate case.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#113Earlier quoted context omitted.
ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case. It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve. As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curi…
For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier. There was multithreading, and…
Is that memory increase seen in peak usage or average/idle usage?
For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#114Earlier quoted context omitted.
They have a PoC, we're arguing about the LLM-generated slop explanation of the PoC.
Is it actually reproducible though and do I want to chase and reproduce something that had no human touch? Do you ?
If it's an actual bug, I hope the Linux devs will. It impacted a human, and it's a bug regardless of human or non-human touch. Of course, looking at a bug is a matter of priority, always has been.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#115Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple us…
Similarly, I've often wondered if this is the root cause of why GitHub has become fragile of late. All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.
And if https://isolveproblems.substack.com/p/how-microsoft-vaporize... is to be believed even in the slightest, any unoptimized paths in Azure's filesystem abstractions could cause usage spikes to have massive splash radius.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#116Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
> I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis
Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human."
Looks like that thread is from...today!
Re: RipGrep musl binaries occasionally segfault during very-large searches
#117Earlier quoted context omitted.
I tried reading and gave up at the "Headline"... Quoting from the bug analysis: >Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault…
This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it. MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses…
Excellent explanation.
I’ve been reading the LWN kernel page for >20 years so I’ve picked up almost all the jargon and could understand it pretty decently.
You’re dead on in saying it was written for that specific audience, not anyone more “normal”.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#118Earlier quoted context omitted.
"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…
So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.
Sounded like a race condition causing a correct new mapping in the TLB to be cleared away by mistake, reverting back to the zero page mapping it was before.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#119Earlier quoted context omitted.
"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actua…
Thanks for the explanation. I should have looked up each term a bit more. So "backing" is the underlying physical memory the page maps to. So if I were to make sense of that piece of slop, is this what it would be? "In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 mem…
The first thread is the user program, the other thread is the kernel doing things on another core.
Reading from the zero page is legal, that doesn’t cause it to crash. The crash is because of a logic “bug” in the program where the zero it reads back causes some other issue in the program.
I say “bug” because it’s clearly impossible without the kernel messing up. It just stored a non-zero value there.
Re: RipGrep musl binaries occasionally segfault during very-large searches
#120Earlier quoted context omitted.
The overflow would still overflow; the use-after-free would still use after free; a musl mask race would still race. Hilarious. Apart from the computer poetry, the conclusion seems to be “it’s something in Linux 7.0 + musl 1.2.5”, although the only reproduction is still on the same physical Threadripper CPU and only sometimes when heavily exercised, so it hasn’t really ruled out a hardware issue.
The computer seems to have identified source code to blame though? Unless of course it hallucinated which there’s always a non zero chance of
It’s the kernel. Nothing a user space library can do should ever be able to call this.
Just happens to be that the musl code is able to hit this and other code isn’t for some reason.