Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust: Investigating an Out of Memory Error

qovery.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Rust: Investigating an Out of Memory Error

#11

Post by erebe__ »

CodesInChaos wrote: post #8

The analysis looks rather half-finished. They did not analyze why so much memory was consumed. If this is the cache which persists after the first call, if it's temporary working memory, or if it's an accumulating memory leak. And why it uses so much memory at all. I couldn't find any other complaints about rust backtrace printing consuming a lot of memory, which I would have expected if this was normal behaviour. So…

Hello,

You can see my other comment https://news.ycombinator.com/item?id=42708904#42756072 for more details.

But yes, the cache does persist after the first call, the resolved symbols stay in the cache to speed up the resolution of next calls.

Regarding the why, it is mainly because

1. this app is a gRPC server and contains a lot of generated code (you can investigate binary bloat with rust with https://github.com/RazrFalcon/cargo-bloat)

2. and that we ship our binary with debug symbols, with those options ``` ENV RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes" ```

For the panic, indeed, I had the same question on Reddit. For this particular service, we don't expect panics at all, it is just that by default we ship all our rust binaries with backtrace enabled. And we have added an extra api endpoint to trigger a catched panic on purpose for other apps to be sure our sizing is correct.

Re: Rust: Investigating an Out of Memory Error

#12

Post by the8472 »

erebe__ wrote: post #9
xgb84j wrote: post #2

In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?

Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…

> 2. we compile the binary with debug symbols

symbols are usually included even with debuglevel 0, unless stripped[0]. And debuginfo is configurable at several levels[1]. If you've set it to 2/full try dropping to a lower level, that might also result in less data to load for the backtrace implementation.

[0] https://users.rust-lang.org/t/difference-between-strip-symbo... [1] https://doc.rust-lang.org/cargo/reference/profiles.html#debu...

Re: Rust: Investigating an Out of Memory Error

#13

Post by erebe__ »

CodesInChaos wrote: post #10
erebe__ wrote: post #9

Earlier quoted context omitted.

Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…

> we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. How big are the uncompressed debug symbols? I'd expected processing uncompressed debug symbols to happen via a memory mapped file, while compressed debug symbols probably need to be extracted to anonymous memory. https://github.com/llvm/llvm-project/issues/63290

Normal build

  cargo build --bin engine-gateway --release
      Finished `release` profile [optimized + debuginfo] target(s) in 1m 00s

  ls -lh target/release/engine-gateway 
  .rwxr-xr-x erebe erebe 198 MB Sun Jan 19 12:37:35 2025    target/release/engine-gateway
what we ship

  export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes" 
  cargo build --bin engine-gateway --release
      Finished `release` profile [optimized + debuginfo] target(s) in 1m 04s

  ls -lh target/release/engine-gateway
  .rwxr-xr-x erebe erebe 61 MB Sun Jan 19 12:39:13 2025  target/release/engine-gateway
The diff is more impressive on some bigger projects

Re: Rust: Investigating an Out of Memory Error

#14

Post by erebe__ »

the8472 wrote: post #12
erebe__ wrote: post #9

Earlier quoted context omitted.

Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…

> 2. we compile the binary with debug symbols symbols are usually included even with debuglevel 0, unless stripped[0]. And debuginfo is configurable at several levels[1]. If you've set it to 2/full try dropping to a lower level, that might also result in less data to load for the backtrace implementation. [0] https://users.rust-lang.org/t/difference-between-strip-symbo... [1] https://doc.rust-lang.org/cargo/reference…

Thanks, was not aware there was granularity for debuginfo ;)

Re: Rust: Investigating an Out of Memory Error

#15

Post by adastra22 »

erebe__ wrote: post #13
CodesInChaos wrote: post #10

Earlier quoted context omitted.

> we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. How big are the uncompressed debug symbols? I'd expected processing uncompressed debug symbols to happen via a memory mapped file, while compressed debug symbols probably need to be extracted to anonymous memory. https://github.com/llvm/llvm-project/issues/63290

Normal build cargo build --bin engine-gateway --release Finished `release` profile [optimized + debuginfo] target(s) in 1m 00s ls -lh target/release/engine-gateway .rwxr-xr-x erebe erebe 198 MB Sun Jan 19 12:37:35 2025  target/release/engine-gateway what we ship export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes" cargo build --bin engine-gateway --release Finished `release`…

The compressed symbols sounds like the likely culprit. Do you really need a small executable? The uncompressed symbols need to be loaded into RAM anyway, and if it is delayed until it is needed then you will have to allocate memory to uncompress them.

Re: Rust: Investigating an Out of Memory Error

#16

Post by erebe__ »

adastra22 wrote: post #15
erebe__ wrote: post #13

Earlier quoted context omitted.

Normal build cargo build --bin engine-gateway --release Finished `release` profile [optimized + debuginfo] target(s) in 1m 00s ls -lh target/release/engine-gateway .rwxr-xr-x erebe erebe 198 MB Sun Jan 19 12:37:35 2025  target/release/engine-gateway what we ship export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes" cargo build --bin engine-gateway --release Finished `release`…

The compressed symbols sounds like the likely culprit. Do you really need a small executable? The uncompressed symbols need to be loaded into RAM anyway, and if it is delayed until it is needed then you will have to allocate memory to uncompress them.

I will give it a shot next week to try out ;P

For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.

Re: Rust: Investigating an Out of Memory Error

#17

Post by delusional »

erebe__ wrote: post #9
xgb84j wrote: post #2

In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?

Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…

> Sorry if the article is misleading.

I don't think the article is misleading, but I do think it's a shame that all the interesting info is saved for this hackernews comment. I think it would make for a more exciting article if you included more of the analysis along with the facts. Remember, as readers we don't know anything about your constraints/system.

Re: Rust: Investigating an Out of Memory Error

#18

Post by xgb84j »

erebe__ wrote: post #9
xgb84j wrote: post #2

In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?

Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…

Thank you for this in-depth reply! Your answer makes a lot of sense. Also thank you for writing the article!

Re: Rust: Investigating an Out of Memory Error

#19

Post by adastra22 »

erebe__ wrote: post #16
adastra22 wrote: post #15

Earlier quoted context omitted.

The compressed symbols sounds like the likely culprit. Do you really need a small executable? The uncompressed symbols need to be loaded into RAM anyway, and if it is delayed until it is needed then you will have to allocate memory to uncompress them.

I will give it a shot next week to try out ;P For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.

Maybe I'm not communicating well. Or maybe I don't understand how the debug symbol compression works at runtime. But my point is that I don't think you are getting the tradeoff you think you are getting. The smaller executable may end up using more RAM. Usually at the deployment stage, that's what matters.

Smaller executables are more for things like reducing distribution sizes, or reducing process launch latency when disk throughput is the issue. When you invoke compression, you are explicitly trading off runtime performance in order to get the benefit of smaller on-disk or network transmission size. For a hosted service, that's usually not a good tradeoff.

Re: Rust: Investigating an Out of Memory Error

#20

Post by prerok »

prerok wrote: post #4
xgb84j wrote: post #2

In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?

Well, based on the article, if there was a memory leak then they should see the steady increase in memory consumption, which was not the case. The only explanation I can see (if their conclusion is accurate) is that the end result of the symbolization is more than 400MB additional memory consumption (which is a lot in my opinion), however the process of the symbolization requires more than 2GB additional memory (whic…

The author replied with additional explanations, so it seems that the additional 400MB were needed because the debug symbols were compressed.
Post reply on HN