Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

71–80 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#71

Post by _3u10 »

dekhn wrote: post #48
underdeserver wrote: post #39

Earlier quoted context omitted.

TIL, unless you explicitly disable memory overcommit, it can and will overcommit. This is crazy to me.

So, uh, how do you feel about fractional reserve banking? Nearly all banks worldwide practice it. Statistically, it's not impossible that the entire world financial system could collapse due to uncorrelated bank runs.

It is impossible. Only a moron of magnificent magnitude would fail to print additional cash to cover the run.

The problems caused by the feds failure to lend to First Bank of America during the Great Depression are well understood by the central banks.

What would likely happen is the overnight rate would go up to 12%, and additional money would be printed to cover withdrawals for the month or two most people would be willing to forgo 12% interest in a potentially inflationary economy.

Re: In C, how do you know if the dynamic allocation succeeded?

#72

Post by jiveturkey »

pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

Most processes that fork know that they are going to fork. Therefore they can pre-fork very early, so as to commit the memory when they only have a bare minimum allocated. Your typical daemon does this anyway.

Some other type of process like an interpreter that can subshell out doesn't know how big the allocation is going to get, would have to pre-fork early on.

In this way, you wouldn't "need" overcommit and the Linux horror of OOM. Well, perhaps you don't need it so badly. Programs that use sparse arrays without mmap() probably need overcommit or lots of swap.

Re: In C, how do you know if the dynamic allocation succeeded?

#73

Post by haxorrr666 »

Nope. Works fine on Fedora 34:
# free -h total used free shared buff/cache available Mem: 31Gi 3.1Gi 2.2Gi 27Mi 25Gi 27Gi Swap: 15Gi 62Mi 15Gi # uname -a Linux athena 5.13.19-200.fc34.x86_64 #1 SMP Sat Sep 18 16:32:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux # gcc -o memaloc memaloc.c # ./memaloc error!

Re: In C, how do you know if the dynamic allocation succeeded?

#74

Post by haxorrr666 »

Nope. Works fine on Fedora 34:

# free -h

               total        used        free      shared  buff/cache   available
Mem: 31Gi 3.1Gi 2.2Gi 27Mi 25Gi 27Gi

Swap: 15Gi 62Mi 15Gi

# uname -a

Linux athena 5.13.19-200.fc34.x86_64 #1 SMP Sat Sep 18 16:32:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# gcc -o memaloc memaloc.c

# ./memaloc

error!

Re: In C, how do you know if the dynamic allocation succeeded?

#75

Post by saagarjha »

joosters wrote: post #62
giomasce wrote: post #59

Earlier quoted context omitted.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on… the flexibility of a separate fork() then exec() means you can…

I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)

Re: In C, how do you know if the dynamic allocation succeeded?

#76

Post by TillE »

one_off_comment wrote: post #16
yjftsjthsd-h wrote: post #3

TLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request. I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).

There's gotta be some way to programmatically determine it, right? It may not be portable. It may require some system calls or something, but there's gotta be a way, right?

The question is what problem are you actually trying to solve.

You can get away with just about anything if you do it in a child worker process which can safely crash.

Re: In C, how do you know if the dynamic allocation succeeded?

#77

Post by saagarjha »

_3u10 wrote: post #10
valleyer wrote: post #8

Earlier quoted context omitted.

Similarly on macOS. There is a limit of 64 gigs in the VM compressor (in-core and on-disk compressed "segments" combined); when this is reached, a process that owns more than 50% of the compressed memory can be killed. See no_paging_space_action() in: https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/ke... Edit: I think the 64 gigs number is out of date -- looks like it's now based in part on the amount of ph…

You can tell the FreeBSD / xnu devs take their job more seriously. A failure in the VM compressor sounds so much more professional than being OOM killed.

This is literally just another way of saying it'll be OOM killed :/

Re: In C, how do you know if the dynamic allocation succeeded?

#78

Post by AshamedCaptain »

yjftsjthsd-h wrote: post #3

TLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request. I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).

Technically the allocation succeeded. That's why you have a pointer and not NULL. So the answer is the classic one: "you check if the pointer is NULL or not".

This entire thing is a wrong question to be asking, anyway.

Even if you are sure right after malloc that "you have the memory 100% available to you", who's not to say that, immediately one nanosecond afterwards, some other process comes in and, well, asks for more memory? The memory you were so sure you had for yourself right after malloc may no longer be there (either swapped out or even discarded if possible), and you will be killed for trying to access it again. Oops.

At no point you can say "I have this memory and it's mine and just mine!" unless you are root and mlock (or equivalents). And I hope you don't do that, anyway. Just let the OS do what it is designed to do.

Re: In C, how do you know if the dynamic allocation succeeded?

#79

Post by scottlamb »

pjc50 wrote: post #51
wahern wrote: post #40

Earlier quoted context omitted.

> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.

How does Solaris handle this? Or Darwin?

I don't use Solaris, but I see they have vfork [1] which should be unaffected. Likewise posix_spawn{,p}. (I think those are often implemented in userspace via vfork; unsure if there's a separate system call on Solaris.)

[1] https://docs.oracle.com/cd/E26505_01/html/816-5167/vfork-2.h...

Re: In C, how do you know if the dynamic allocation succeeded?

#80

Post by SV_BubbleTime »

edwinbalani wrote: post #19
davidw wrote: post #14

This is why some embedded systems don't do malloc (IIRC...been a while since I've read much about those).

That's right - some "safe" coding standards like MISRA C go as far as forbidding use of malloc() without explicit justification. If you still need dynamic allocation, you might choose to have a custom allocator working from a fixed-size pool or arena created in code (possibly itself carved out of RAM with malloc(), but importantly only once at first setup, where you have a better guarantee that the allocation will su…

All good stuff.

I’m working on a project now and trying my best to make it MISRA compatible because FreeRTOS did the same. You don’t use malloc() but rather pvPortMalloc() which pulls from an already allocated pool. Their HEAP4 system tries to keep the chunks organized. So far I’ve been very happy with it.

Post reply on HN