Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

131–140 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#131

Post by tsimionescu »

lmilcin wrote: post #37
yjftsjthsd-h wrote: post #33

Earlier quoted context omitted.

No, it's worse than that - the answer is "yes", because virtual memory + overcommit means that most of the time the OS will happily allow you to allocate more memory than physical+swap, and essentially gamble that you won't actually need all of it (and this is implemented because apparently that's almost always a winning bet).

Yeah. And the issue is that the actual problem happens sometime later when the application actually tries to use that memory. So you replaced an error that is relatively simple to handle with something that is impossible to handle reliably. So the operating system very much doesn't like to admit it doesn't have physical memory to back the area you are trying to use. Now it does not have a simple way to signal this to…

Interestingly, Linux's OOMKiller actually gives you (the sysadmin or system designer) more control over what happens when the system is low on memory than disabling overcommit.

In a system without overcommit, every process is taking memory out of the shared pool, until some random process is the unlucky one that can't allocate more. In a happy case, that unlucky process also has some data that it can let go of. But this is entirely random - you could have a bunch of gigabyte-sized application caches in half of your processes, but the NTP daemon might be the one who cends up failing because it can't allocate a few more bytes. Even worse, it could be the SSH server or bash failing to spawn a new shell, preventing any kind of intervention on the system.

With OOMKiller, you can at least define some priorities, and ensure some critical processes are never going to be killed or stalled.

Re: In C, how do you know if the dynamic allocation succeeded?

#132

Post by tsimionescu »

naniwaduni wrote: post #125
tsimionescu wrote: post #121

Earlier quoted context omitted.

> Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). This doesn't work because of the horrible fork/exec design. If I am a huge process and I want to run `ls`, I will first have to clone myself using fork(), which may trigger an OOM, even if the first action of my clone would have been exec(), ignoring all of that memory. I am always surprised that no one has added a sane 's…

We do have posix_spawn &c., which are expressly designed to be implementable even in situations where fork doesn't make sense.

Interesting, I'd never learned about that. If I understand correctly, this actually used to be implemented as a fork()+exec() on many systems, though. It's actually vfork() that addressed the problem and allowed an actually useful implementation of posix_spawn in Linux at least.

Re: In C, how do you know if the dynamic allocation succeeded?

#134

Post by jart »

marcan_42 wrote: post #110
saagarjha wrote: post #75

Earlier quoted context omitted.

I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)

Are there any things that are illegal between fork and exec? It is perfectly legal to exec whenever you want, and it it perfectly legal to fork whenever you want. I'm not aware of any requirements around the fork/exec sequence.

There's no restrictions in that sense. However the computer might not do what you intended if you were to do something like fputc('x',stdout);fork();fflush(0);

Re: In C, how do you know if the dynamic allocation succeeded?

#135

Post by jart »

nextaccountic wrote: post #116
saagarjha wrote: post #75

Earlier quoted context omitted.

I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)

I think you mean between vfork and exec. The Linux manpage says: Standard description (From POSIX.1) The vfork() function has the same effect as fork(2), except that the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function be…

Note that modern POSIX recommends just calling fork() because "[vfork] was previously under-specified". The glibc man pages shouldn't say undefined since that's too strong a word. There is so much you can actually do in practice during vfork(). Pretty much all system calls are just as safe. It's mostly a question of whether or not your userspace libraries might clobber some global, due to how vfork puts the process in a state where implementation details that are normally private become part of the public surface area of these apis, and not many library authors are disposed to offer those kinds of assurances.

Re: In C, how do you know if the dynamic allocation succeeded?

#137

Post by Dylan16807 »

tsimionescu wrote: post #128
OldHand2018 wrote: post #57

Earlier quoted context omitted.

Let's say you malloc some memory and the computer actually has everything available. Everything is great, up until some other process on your system does a fork bomb of an infinitely recursive program that allocates nothing on the heap. You've just got a whole lot of quickly growing stacks hoovering up your physical memory pages.

Stack vs Heap is a userspace level concept. As far as the kernel memory manager knows, it's all just allocated memory - some is allocated by the process- or thread-spawning routines, some is allocated by malloc(), but they're using the exact same pool. If overcommit is disabled and someone has allocated most system memory, fork() and exec() and pthread_create() etc. will theoretically fail with ENOMEM. A bigger probl…

It's really unfortunate that you can't set minimum and maximum disk cache on Linux.

Re: In C, how do you know if the dynamic allocation succeeded?

#138

Post by bscphil »

dahart wrote: post #124
Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

But this feature is also why malloc on Windows can take a long time to return - seconds or more if you malloc too much. I recently bumped into this problem on the GPU. Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory, a malloc on the GPU when running low on space can end up searching & even defragging virtual memory to satisfy the request. Crazy! Just…

> Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory

Wow, this seems really unintuitive to me, especially on a OS that doesn't overcommit. Is it as unintuitive as it sounds, or is there a good reason for this?

Re: In C, how do you know if the dynamic allocation succeeded?

#139

Post by trhway »

angry_octet wrote: post #117
trhway wrote: post #43

Earlier quoted context omitted.

>there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now. Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people…

Virtual memory is always on... If you actually hit swap the system effectively deadlocks anyway because the swap daemon is too dumb by being too fair. Persistent memory might change this but it needs so much work.

it was a throwback down the memory lane and the years before my first Linux in 1995 i had been dealing with Windows where configuration of swap was "configuring virtual memory", so i automatically used that term.

> If you actually hit swap the system effectively deadlocks anyway

that depends. In many cases in the past the options would be either with swap and thus slow or pretty much not at all. And again these days there is so much memory that there is always a way to avoid the swapping. Though i've met funny situations in recent years like when a several terabyte sized database process would get killed by the OOMKiller on a machine with overcommit left on and no swap configured.

Re: In C, how do you know if the dynamic allocation succeeded?

#140

Post by dahart »

bscphil wrote: post #138
dahart wrote: post #124

Earlier quoted context omitted.

But this feature is also why malloc on Windows can take a long time to return - seconds or more if you malloc too much. I recently bumped into this problem on the GPU. Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory, a malloc on the GPU when running low on space can end up searching & even defragging virtual memory to satisfy the request. Crazy! Just…

> Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory Wow, this seems really unintuitive to me, especially on a OS that doesn't overcommit. Is it as unintuitive as it sounds, or is there a good reason for this?

There are good reasons, not all of which I know or understand, but my basic mental model is that the backing is required for various scenarios where virtual memory might need to get relocated, either in response to a malloc or to something else. I’m told it’s complicated by having the GPU drive the display, and by scenarios involving multiple GPUs in a system.

I’m guessing the description in the WDDM docs about DDI might be relevant to what’s happening under the covers in WDDM2? I’m not sure… :) https://docs.microsoft.com/en-us/windows-hardware/drivers/di...

Post reply on HN