In C, how do you know if the dynamic allocation succeeded?
1–10 of 198 posts
Re: In C, how do you know if the dynamic allocation succeeded?
#2I'm a little disappointed that the article didn't answer the question, or at least try to. A discussion of using read/write vs mincore vs trying to catch a SIGSEGV would've been a nice addition.
Re: In C, how do you know if the dynamic allocation succeeded?
#3TLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request.
I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).
Re: In C, how do you know if the dynamic allocation succeeded?
#4This difference between "malloc() succeeded and physical/swap memory is actually available" also has somewhat corresponding impact on how you measure memory usage.
One approach is RSS, the memory in physical RAM... but what if you're swapping? Then again, maybe you swapped out memory you don't actually need and ignoring swap is fine.
The other approach is "how much memory you allocated", and then you hit fun issues mentioned in this article, like "the OS doesn't actually _really_ allocate until you touch the page".
(Longer version: https://pythonspeed.com/articles/measuring-memory-python/)
Re: In C, how do you know if the dynamic allocation succeeded?
#5It’s likely being OOM killed on Linux. Not sure what’s happening on Mac. Try allocating a terabyte of swap and it should run.
Alternatively use mmap & mlock to verify the allocation succeeded, but the process can still be OOM killed at any time for any reason.
Re: In C, how do you know if the dynamic allocation succeeded?
#6I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.
Re: In C, how do you know if the dynamic allocation succeeded?
#7This has nothing to do with C. The problem/feature is with the systemcall that does the allocation, and any language has to use it.
Re: In C, how do you know if the dynamic allocation succeeded?
#8It’s likely being OOM killed on Linux. Not sure what’s happening on Mac. Try allocating a terabyte of swap and it should run. Alternatively use mmap & mlock to verify the allocation succeeded, but the process can still be OOM killed at any time for any reason.
Similarly on macOS. There is a limit of 64 gigs in the VM compressor (in-core and on-disk compressed "segments" combined); when this is reached, a process that owns more than 50% of the compressed memory can be killed.
See no_paging_space_action() in:
https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/ke...
Edit: I think the 64 gigs number is out of date -- looks like it's now based in part on the amount of physical memory in the machine.
Re: In C, how do you know if the dynamic allocation succeeded?
#9On Linux, people should be pointed to
https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
Re: In C, how do you know if the dynamic allocation succeeded?
#10It’s likely being OOM killed on Linux. Not sure what’s happening on Mac. Try allocating a terabyte of swap and it should run. Alternatively use mmap & mlock to verify the allocation succeeded, but the process can still be OOM killed at any time for any reason.
Similarly on macOS. There is a limit of 64 gigs in the VM compressor (in-core and on-disk compressed "segments" combined); when this is reached, a process that owns more than 50% of the compressed memory can be killed. See no_paging_space_action() in: https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/ke... Edit: I think the 64 gigs number is out of date -- looks like it's now based in part on the amount of ph…
You can tell the FreeBSD / xnu devs take their job more seriously. A failure in the VM compressor sounds so much more professional than being OOM killed.