Rust: Investigating an Out of Memory Error
1–10 of 32 posts
Re: Rust: Investigating an Out of Memory Error
#2In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory.
Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
Re: Rust: Investigating an Out of Memory Error
#3In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
[flagged]
Re: Rust: Investigating an Out of Memory Error
#4In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
Well, based on the article, if there was a memory leak then they should see the steady increase in memory consumption, which was not the case.
The only explanation I can see (if their conclusion is accurate) is that the end result of the symbolization is more than 400MB additional memory consumption (which is a lot in my opinion), however the process of the symbolization requires more than 2GB additional memory (which is incredibly a lot).
Re: Rust: Investigating an Out of Memory Error
#5Can't they print something that llvm-symbolizer would pick up offline?
Re: Rust: Investigating an Out of Memory Error
#6I once had a faulty python based ai image generator running on my machine that used all 64 gigs of ram and oomed with a memory dump written to fs. This is no fun when that happens. But mostly these kind of bugs are misconfigurations or bad code, never ending while loops, whatever.
Re: Rust: Investigating an Out of Memory Error
#7In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
I don't think the 4 GiB instance actually ran into an OOM error. They merely observed a 400 MiB memory spike. The crashing instances were limited to 256 and later 512 MiB.
(Assuming that the article incorrectly used Mib when they meant MiB. Used correctly b=bit, B=byte)
Re: Rust: Investigating an Out of Memory Error
#8The analysis looks rather half-finished. They did not analyze why so much memory was consumed. If this is the cache which persists after the first call, if it's temporary working memory, or if it's an accumulating memory leak. And why it uses so much memory at all.
I couldn't find any other complaints about rust backtrace printing consuming a lot of memory, which I would have expected if this was normal behaviour. So I wonder if there is anything special about their environment or usecase?
I would assume that the same OOM problem would arise when printing a panic backtrace. Either their instance has enough memory to print backtraces, or it doesn't. So I don't understand why they only disable lib backtraces.
Re: Rust: Investigating an Out of Memory Error
#9In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
Sorry if the article is misleading.
The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting.
Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling.
Regarding the increase of 400MiB, yeah it is a lot, and it was a surprise to us too. We were not expecting such increase. There are, I think 2 reasons behind this.
1. This service is a grpc server, which has a lot of code generated, so lots of symbols
2. we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. Which may part be of this issue.
Re: Rust: Investigating an Out of Memory Error
#10In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?
Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a…
> we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary.
How big are the uncompressed debug symbols? I'd expected processing uncompressed debug symbols to happen via a memory mapped file, while compressed debug symbols probably need to be extracted to anonymous memory.