great one! and i would recommnd this hands-on guide for diagnosing memory leaks in Rust applications. it explains how to enable heap profiling in 'jemalloc', collect memory allocation data, and generate flame graphs for analysis. https://greptime.com/blogs/2024-01-18-memory-leak#diagnosing...
Rust: Investigating an Out of Memory Error
21–30 of 32 posts
Re: Rust: Investigating an Out of Memory Error
#22Earlier quoted context omitted.
The compressed symbols sounds like the likely culprit. Do you really need a small executable? The uncompressed symbols need to be loaded into RAM anyway, and if it is delayed until it is needed then you will have to allocate memory to uncompress them.
I will give it a shot next week to try out ;P For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.
Container images use compression too, so having the debug section uncompressed shouldn't actually make the images any bigger.
Re: Rust: Investigating an Out of Memory Error
#23Reminds me a bit of a post my colleague wrote a while back: https://www.svix.com/blog/heap-fragmentation-in-rust-applica...
Re: Rust: Investigating an Out of Memory Error
#24Ahh, I did this in Python before I learned about Cursor and Sourceforge‘s Cody. I‘d use a template where I provide a tree of my project structure, and then put code file contents in my template file, and then have a full repo in one giant markdown file. This only worked for smaller projects, but it worked damn well to provide the full context to an LLM to then ask questions about my code :)
Re: Rust: Investigating an Out of Memory Error
#25Earlier quoted context omitted.
I will give it a shot next week to try out ;P For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.
Maybe I'm not communicating well. Or maybe I don't understand how the debug symbol compression works at runtime. But my point is that I don't think you are getting the tradeoff you think you are getting. The smaller executable may end up using more RAM. Usually at the deployment stage, that's what matters. Smaller executables are more for things like reducing distribution sizes, or reducing process launch latency whe…
It is most likely me reading too quickly. I was caught off guard by the article gaining traction in a Sunday, and as I have other duties during the weekend, I am reading/responding only when I can sneak in.
For your comment, I think you are right regarding compression of debug symbols that add up to the peak memory, but I think you are misleading when you think the debug symbols are uncompressed when the app/binary is started/loaded. Decompression only happens for me when this section is accessed by debugger or equivalent. It is not the same thing as when the binary is fully compressed, like with upx for example.
I have done a quick sanity check on my desktop, I got.
[profile.release]
lto = "thin"
debug = true
strip = false
export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes"
cargo build --bin engine-gateway --release
When compiled with those flags
export RUSTFLAGS="-C force-frame-pointers=yes"
cargo build --bin engine-gateway --release
So the peak is taller indeed when the debug section is compressed, but the binary in memory when started is roughly equivalent. (virtual mem too)
I had some hard time getting a source that may validate my belief regarding when the debug symbol are uncompressed. But based on https://inbox.sourceware.org/binutils/20080622061003.D279F3F... and the help of claude.ai, I would say it is only when those sections are accessed.
for what is worth, the whole answer of claude.ai
The debug sections compressed with --compress-debug-sections=zlib are decompressed:
At runtime by the debugger (like GDB) when it needs to access the debug information:
When setting breakpoints
When doing backtraces
When inspecting variables
During symbol resolution
When tools need to read debug info:
During coredump analysis
When using tools like addr2line
During source-level debugging
When using readelf with the -w option
The compression is transparent to these tools - they automatically handle the decompression when needed. The sections remain compressed on disk, and are only decompressed in memory when required.
This helps reduce the binary size on disk while still maintaining full debugging capabilities, with only a small runtime performance cost when the debug info needs to be accessed.
The decompression is handled by the libelf/DWARF libraries that these tools use to parse the ELF files.Re: Rust: Investigating an Out of Memory Error
#26Can't they print something that llvm-symbolizer would pick up offline?
Yes, that is typically the way to go.
Collecting a call stack only requires unwinding information (which is usually already present for C++ exceptions / Rust panics), not full debug symbols. This gives you a list of instruction pointers. (on Linux, the glibc `backtrace` function can help with this)
Print those instruction pointers in a relative form (e.g. "my_binary+0x1234") so that the output is independent of ASLR.
The above is all that needs to happen on the production/customer machines, so you don't need to ship debug symbols -- you can ship `strip`ped binaries.
On your own infrastructure, keep the original un-stripped binaries around. We use a script involving elfutil's eu-addr2line with those original binaries to turn the module+relative_address stack trace into a readable symbolized stack trace. I wasn't aware of llvm-symbolizer yet, seems like that can do the same job as eu-addr2line. (There's also binutil's addr2line but in my experience that didn't work as well as eu-addr2line)
Re: Rust: Investigating an Out of Memory Error
#27Earlier quoted context omitted.
Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…
> Sorry if the article is misleading. I don't think the article is misleading, but I do think it's a shame that all the interesting info is saved for this hackernews comment. I think it would make for a more exciting article if you included more of the analysis along with the facts. Remember, as readers we don't know anything about your constraints/system.
It was a parti pris by me, I wanted the article to stay focus on the how, not much on the why. But I agree, even while the context is specific to us, many people wanted more interest of the surrounding, and why it happened. I wanted to explain the method ¯\_(ツ)_/¯
Re: Rust: Investigating an Out of Memory Error
#28Reminds me a bit of a post my colleague wrote a while back: https://www.svix.com/blog/heap-fragmentation-in-rust-applica...
Yeap, just recently discovered that pretty much any long living app in Rust should switch jemalloc for this reason.
Re: Rust: Investigating an Out of Memory Error
#29Ahh, I did this in Python before I learned about Cursor and Sourceforge‘s Cody. I‘d use a template where I provide a tree of my project structure, and then put code file contents in my template file, and then have a full repo in one giant markdown file. This only worked for smaller projects, but it worked damn well to provide the full context to an LLM to then ask questions about my code :)
[deleted]
Re: Rust: Investigating an Out of Memory Error
#30Earlier quoted context omitted.
Maybe I'm not communicating well. Or maybe I don't understand how the debug symbol compression works at runtime. But my point is that I don't think you are getting the tradeoff you think you are getting. The smaller executable may end up using more RAM. Usually at the deployment stage, that's what matters. Smaller executables are more for things like reducing distribution sizes, or reducing process launch latency whe…
It is most likely me reading too quickly. I was caught off guard by the article gaining traction in a Sunday, and as I have other duties during the weekend, I am reading/responding only when I can sneak in. For your comment, I think you are right regarding compression of debug symbols that add up to the peak memory, but I think you are misleading when you think the debug symbols are uncompressed when the app/binary i…
Thanks for running these checks. I’m learning from this too!