Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

101–110 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#101

Post by toast0 »

milkey_mouse wrote: post #88
Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

I've found overcommit useful for some scientific computing applications, but you're right it's not the most intuitive default. You can disable memory overcommit on Linux for the same behavior as Windows, if you want. https://www.kernel.org/doc/Documentation/vm/overcommit-accou... Also, I found this post that suggests Windows technically does overcommit memory, but only for stacks(‽): https://superuser.com/questions/1…

Disabling overcommit on Linux doesn't tend to enable library and utility developers to have used memory responsibly. So it can be hard to run a system with common software.

Re: In C, how do you know if the dynamic allocation succeeded?

#102

Post by scottlamb »

pcwalton wrote: post #87
scottlamb wrote: post #83

Earlier quoted context omitted.

> Vfork is usually a better solution. I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions." So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In…

No-MMU Linux usually just disables fork and uses vfork instead. The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI).

> The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI).

Interesting, thanks. That didn't come up on our system—not only did we not use shared libraries but we also linked the whole system into one binary, kernel and all. Link times were atrocious in combination with identical code folding and big VLIW sentences, but it worked.

Re: In C, how do you know if the dynamic allocation succeeded?

#103

Post by bronzeage »

This is actually not the common behavior you'll see when normally exceeding memory limits. If you allocate a lot of memory with relatively small allocations, you'll actually reach a null malloc. It's only when asking for a huge allocation all at once that this behavior happens.

Re: In C, how do you know if the dynamic allocation succeeded?

#104

Post by toast0 »

itamarst wrote: post #4

This difference between "malloc() succeeded and physical/swap memory is actually available" also has somewhat corresponding impact on how you measure memory usage. One approach is RSS, the memory in physical RAM... but what if you're swapping? Then again, maybe you swapped out memory you don't actually need and ignoring swap is fine. The other approach is "how much memory you allocated", and then you hit fun issues m…

> how you measure memory usage.

Facebook put out something about measuring memory use on Linux after I left, and I haven't looked at it... But the best way I've seen is to have swap of size min(512M, 2x RAM) and measure the usage of that. There's some cases where sometimes something bigish gets swapped and you're actually fine, but often you really want to address that anyway.

Re: In C, how do you know if the dynamic allocation succeeded?

#105

Post by magicalhippo »

enedil wrote: post #95
magicalhippo wrote: post #84

Earlier quoted context omitted.

> But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy th…

That also means that if you used seccomp to restrict mprotect syscall, the child process may never ever write again, which is an unfortunate design choice.

Seccomp can check for the PROT_EXEC flag though, and let the others pass, no?

Re: In C, how do you know if the dynamic allocation succeeded?

#106

Post by willbudd »

shoo wrote: post #94
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

I first learned about this while trying to understand processes being killed by OOM in production. We had python 2.x batch jobs being executed by long-running python worker processes -- some of the arbitrary application code in some of the arbitrary batch jobs would occasionally want to execute some command line tool in a new process, and to create the new process under the hood python's subprocess library would fork…

Sounds to me like you (or Python devs?) should be looking at clone(), which is basically a lower level version of fork() that provides precise control over every detail with which the new process is created, including what (if any) parent memory you want to share.

The manual page is quite informative: man clone

Re: In C, how do you know if the dynamic allocation succeeded?

#107

Post by ungamedplayer »

convolvatron wrote: post #44
kibwen wrote: post #21

Earlier quoted context omitted.

The answer is that you kind of can't. You're at the mercy of the OS to give you accurate information, and malloc as an interface isn't set up to distinguish between virtual memory and "actual" memory. We could imagine a separate interface that would allow the OS to communicate this distinction (or hacks like you allude to), but I don't know of any standard approach.

before linux - every unix OS would fail a page allocation if there wasn't a backing store. full stop. this worked really well

Until it didn't.

Re: In C, how do you know if the dynamic allocation succeeded?

#108

Post by shoo »

willbudd wrote: post #106
shoo wrote: post #94

Earlier quoted context omitted.

I first learned about this while trying to understand processes being killed by OOM in production. We had python 2.x batch jobs being executed by long-running python worker processes -- some of the arbitrary application code in some of the arbitrary batch jobs would occasionally want to execute some command line tool in a new process, and to create the new process under the hood python's subprocess library would fork…

Sounds to me like you (or Python devs?) should be looking at clone(), which is basically a lower level version of fork() that provides precise control over every detail with which the new process is created, including what (if any) parent memory you want to share. The manual page is quite informative: man clone

it looks like cpython is improving and cpython 3.10 now uses vfork when launching subprocesses in linux

years ago when i first hit this in production, we ended up working around it by rewriting our application code to use a pure-python library that did the equivalent thing as the separate command line tool we were trying to launch.

Re: In C, how do you know if the dynamic allocation succeeded?

#110

Post by marcan_42 »

saagarjha wrote: post #75
joosters wrote: post #62

Earlier quoted context omitted.

It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on… the flexibility of a separate fork() then exec() means you can…

I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)

Are there any things that are illegal between fork and exec? It is perfectly legal to exec whenever you want, and it it perfectly legal to fork whenever you want. I'm not aware of any requirements around the fork/exec sequence.
Post reply on HN