This not "In C", this is a system-related issue. I always disable overcommit as I see no benefit on my systems.
In C, how do you know if the dynamic allocation succeeded?
11–20 of 198 posts
Re: In C, how do you know if the dynamic allocation succeeded?
#12I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.
Not the same thing.
malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.
Re: In C, how do you know if the dynamic allocation succeeded?
#13One of my interview questions starts with "Can a program allocate more memory than is physically available on the server?"
Everybody gets this wrong (which is funny for a binary question) but it starts an interesting discussion through which I hope to learn how much they know about OS and virtual memory.
Re: In C, how do you know if the dynamic allocation succeeded?
#14This is why some embedded systems don't do malloc (IIRC...been a while since I've read much about those).
Re: In C, how do you know if the dynamic allocation succeeded?
#15It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being necessary pretty quickly. Otherwise, fork/exec would stop working in any process using more than half of the system memory.
Re: In C, how do you know if the dynamic allocation succeeded?
#16TLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request. I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).
There's gotta be some way to programmatically determine it, right? It may not be portable. It may require some system calls or something, but there's gotta be a way, right?
Re: In C, how do you know if the dynamic allocation succeeded?
#17It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…
wouldn't you just account the COW pages against the parent until they are copied?
kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill.
this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now.
Re: In C, how do you know if the dynamic allocation succeeded?
#18This not "In C", this is a system-related issue. I always disable overcommit as I see no benefit on my systems.
I used to disable overcommit, but I ran into a bunch of compilers that would crash because of it despite having 30GiB of free (as reported by free(1) ) memory available.
Re: In C, how do you know if the dynamic allocation succeeded?
#19This is why some embedded systems don't do malloc (IIRC...been a while since I've read much about those).
That's right - some "safe" coding standards like MISRA C go as far as forbidding use of malloc() without explicit justification.
If you still need dynamic allocation, you might choose to have a custom allocator working from a fixed-size pool or arena created in code (possibly itself carved out of RAM with malloc(), but importantly only once at first setup, where you have a better guarantee that the allocation will succeed).
Re: In C, how do you know if the dynamic allocation succeeded?
#20This has nothing to do with C. The problem/feature is with the systemcall that does the allocation, and any language has to use it.
I pooped and scooped the poop said the Malloy to me be free. Along came the scoop and scoop it went. Free it did. In the bucket it tumbled for the others to enjoy.