Handling out of memory conditions in C
eli.thegreenplace.net
Handling out of memory conditions in C
1–10 of 15 posts
Re: Handling out of memory conditions in C
#2Perhaps I'm missing this, but what about the "return failure" policy (for libraries)? If malloc fails, log an error and return a failure to the calling application. It isn't recovery, since it doesn't attempt to continue working, but it doesn't abort() the whole program (including calling application) either.
The calling application can then close the current handle to our library if it chooses (thus freeing up all of the memory again).
(This is what our library, x264, does, and IMO makes the most sense when dealing with situations where it would be insane to try to recovery, but in which one doesn't want to kill the calling app as well. We added malloc checking when it became possible to set options that would hit 2GB allocation on a 32-bit system.)
Re: Handling out of memory conditions in C
#3The "segfault policy" is an extremely bad idea.
Over the past couple years, several major security vulnerabilities have been caused by programs that took the "segfault" path making attacker-controlled array references to NULL pointers. It's not hard to see that if you index something with a standard native integer, 0 + index can mean any word in memory. If you write to that, that's game over.
It's for a similar reason that I don't recommend developers explicitly check malloc, or use xmalloc wrappers. Both are opportunities to make an error that shouldn't be possible in properly deployed code. Adobe Flash had explicit error checking with a convoluted O-O-M policy, forgot a check, and Mark Dowd was able to use a Flash header field to create a 32 bit offset to a NULL pointer, and:
http://chargen.matasano.com/chargen/2007/7/3/this-new-vulner...
xmalloc addresses this, except if you forget to use xmalloc, or if you forget that "strdup" needs an "x" too, or if you use a library that calls malloc.
The safest approach is just to configure malloc to abort (or invoke your O-O-M handler via longjmp or signal). Most C libraries have explicit support for this, but every development platform will allow you to preload a malloc wrapper if they don't.
For what it's worth, I came by this opinion not because of security, but because as a systems programmer I was tired of dealing with useless conditionals on a function that's called so often; at one job, I shrank my codebase by 30% (!) by losing malloc checks and the associated "this function should return -1 to indicate malloc failed inside of it" detritus it carried. The "just preload a malloc that aborts" rationale I got from a friend (then) at Juno.
Re: Handling out of memory conditions in C
#4The "segfault policy" is an extremely bad idea. Over the past couple years, several major security vulnerabilities have been caused by programs that took the "segfault" path making attacker-controlled array references to NULL pointers. It's not hard to see that if you index something with a standard native integer, 0 + index can mean any word in memory. If you write to that, that's game over. It's for a similar reaso…
While I agree the "segfault policy" is a bad idea, if you've got dedicated hardware for checking invalid addresses, it seems to me that you should use it.
After all, checking the null pointer is still slow.
Combining a malloc wrapper with the segfault policy is a great way to go, for example:
void*my_malloc(int s){int *x=malloc(s);*x=0;return(void*)x;}Re: Handling out of memory conditions in C
#5The "segfault policy" is an extremely bad idea. Over the past couple years, several major security vulnerabilities have been caused by programs that took the "segfault" path making attacker-controlled array references to NULL pointers. It's not hard to see that if you index something with a standard native integer, 0 + index can mean any word in memory. If you write to that, that's game over. It's for a similar reaso…
While I agree the "segfault policy" is a bad idea, if you've got dedicated hardware for checking invalid addresses, it seems to me that you should use it. After all, checking the null pointer is still slow. Combining a malloc wrapper with the segfault policy is a great way to go, for example: void*my_malloc(int s){int *x=malloc(s);*x=0;return(void*)x;}
Isn't the branch in that NULL check pretty much the most predictable branch in your program?
Re: Handling out of memory conditions in C
#6Earlier quoted context omitted.
While I agree the "segfault policy" is a bad idea, if you've got dedicated hardware for checking invalid addresses, it seems to me that you should use it. After all, checking the null pointer is still slow. Combining a malloc wrapper with the segfault policy is a great way to go, for example: void*my_malloc(int s){int *x=malloc(s);*x=0;return(void*)x;}
Isn't the branch in that NULL check pretty much the most predictable branch in your program?
And furthermore, if you're calling malloc enough to make checking your mallocs have a significant overhead, you're doing it horribly wrong.
Re: Handling out of memory conditions in C
#7The "segfault policy" is an extremely bad idea. Over the past couple years, several major security vulnerabilities have been caused by programs that took the "segfault" path making attacker-controlled array references to NULL pointers. It's not hard to see that if you index something with a standard native integer, 0 + index can mean any word in memory. If you write to that, that's game over. It's for a similar reaso…
The article says:
"Why abort with an error message, when a segmentation fault would do? With a segfault, we can at least inspect the code dump and find out where the fault was?"
abort also produces a core dump, so this is a bad idea for that reason too.
Re: Handling out of memory conditions in C
#8Firstly, I've never had a malloc fail because I code very defensively.
If memory is likely to be an issue, you should control the usage of your memory to start with.
Also, the virtual address space is usually bigger than physical memory (on non-crap OS's and libs) so the performance of the memory access (when swap is involved) is an issue before you get to the stage of running out of memory. Last thing you want is MMU faults and time in the kernel to worry about.
Re: Handling out of memory conditions in C
#9The "segfault policy" is an extremely bad idea. Over the past couple years, several major security vulnerabilities have been caused by programs that took the "segfault" path making attacker-controlled array references to NULL pointers. It's not hard to see that if you index something with a standard native integer, 0 + index can mean any word in memory. If you write to that, that's game over. It's for a similar reaso…
The article says: "Why abort with an error message, when a segmentation fault would do? With a segfault, we can at least inspect the code dump and find out where the fault was?" abort also produces a core dump, so this is a bad idea for that reason too.
[deleted]
Re: Handling out of memory conditions in C
#10Very good article, I like the examination of various open source project's use of memory allocation and failure checking.
For me personally it depends on the project. If there is nothing to clean up in the rest of the code an abort in the malloc function is fine. But if there are other things to deal with at shutdown you have to bit the bullet and check your return values all the way back up the call stack.