Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

21–30 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#21

Post by kibwen »

mirashii wrote: post #2

I'm a little disappointed that the article didn't answer the question, or at least try to. A discussion of using read/write vs mincore vs trying to catch a SIGSEGV would've been a nice addition.

The answer is that you kind of can't. You're at the mercy of the OS to give you accurate information, and malloc as an interface isn't set up to distinguish between virtual memory and "actual" memory. We could imagine a separate interface that would allow the OS to communicate this distinction (or hacks like you allude to), but I don't know of any standard approach.

Re: In C, how do you know if the dynamic allocation succeeded?

#22

Post by Spivak »

lmilcin wrote: post #12
dekhn wrote: post #6

I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

Not the same thing. malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.

That's exactly what fractional reserve banking is.

Re: In C, how do you know if the dynamic allocation succeeded?

#23

Post by edwinbalani »

edwinbalani wrote: post #19
davidw wrote: post #14

This is why some embedded systems don't do malloc (IIRC...been a while since I've read much about those).

That's right - some "safe" coding standards like MISRA C go as far as forbidding use of malloc() without explicit justification. If you still need dynamic allocation, you might choose to have a custom allocator working from a fixed-size pool or arena created in code (possibly itself carved out of RAM with malloc(), but importantly only once at first setup, where you have a better guarantee that the allocation will su…

(All that said, embedded systems sometimes don't have virtual memory, so the original problem stated in the link is just not a thing...)

> working from a fixed-size pool or arena created in code (possibly itself carved out of RAM with malloc(), but importantly only once at first setup, where you have a better guarantee that the allocation will succeed)

And I should add to this that you probably want to access all of the pool/arena to do setup, or just ensure it's physically allocated if you are running in a virtual memory space. This is something that is reasonable at setup time, though.

Re: In C, how do you know if the dynamic allocation succeeded?

#24

Post by floxy »

lmilcin wrote: post #13

One of my interview questions starts with "Can a program allocate more memory than is physically available on the server?" Everybody gets this wrong (which is funny for a binary question) but it starts an interesting discussion through which I hope to learn how much they know about OS and virtual memory.

Don't leave us hanging. The "obvious" answer would seem to be "yes" because of swap. But if everyone gets that wrong...

Re: In C, how do you know if the dynamic allocation succeeded?

#25

Post by secondcoming »

convolvatron wrote: post #17
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.

Re: In C, how do you know if the dynamic allocation succeeded?

#26

Post by lmilcin »

Spivak wrote: post #22
lmilcin wrote: post #12

Earlier quoted context omitted.

Not the same thing. malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.

That's exactly what fractional reserve banking is.

Funny thing, you are right. I was thinking about something else I guess.

Re: In C, how do you know if the dynamic allocation succeeded?

#27

Post by secondcoming »

pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

Even the linux docs for it suggest to turn it on if you're working with sparse arrays, there's no mention of fork()

https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

Re: In C, how do you know if the dynamic allocation succeeded?

#28

Post by jimbob45 »

andi999 wrote: post #7

This has nothing to do with C. The problem/feature is with the systemcall that does the allocation, and any language has to use it.

Is your issue that the system call doesn't return enough diagnostic information? If so, how would you have done it differently? I'm asking out of curiosity, not out of a reflexive instinct to defend C (which has many problems).

Re: In C, how do you know if the dynamic allocation succeeded?

#29

Post by zeusk »

lmilcin wrote: post #12
dekhn wrote: post #6

I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

Not the same thing. malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.

What do you think is a bank run?

Re: In C, how do you know if the dynamic allocation succeeded?

#30

Post by wmf »

one_off_comment wrote: post #16
yjftsjthsd-h wrote: post #3

TLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request. I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).

There's gotta be some way to programmatically determine it, right? It may not be portable. It may require some system calls or something, but there's gotta be a way, right?

I guess you could mlock() your memory and see if that succeeds or fails. When overcommit is enabled no memory is really safe.
Post reply on HN