Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

111–120 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#111

Post by marcosdumay »

convolvatron wrote: post #44
kibwen wrote: post #21

Earlier quoted context omitted.

The answer is that you kind of can't. You're at the mercy of the OS to give you accurate information, and malloc as an interface isn't set up to distinguish between virtual memory and "actual" memory. We could imagine a separate interface that would allow the OS to communicate this distinction (or hacks like you allude to), but I don't know of any standard approach.

before linux - every unix OS would fail a page allocation if there wasn't a backing store. full stop. this worked really well

It has always lead to all kinds of problems, like computers becoming inaccessible, and corrupted data because of segfaults between disk accesses.

And also nobody ever checked the malloc result anyway. Competent programmers just ensured that the segfault wasn't a huge problem. So at the best of the days, all it did was reducing the capacity of the computer.

Re: In C, how do you know if the dynamic allocation succeeded?

#112

Post by foobarian »

floxy wrote: post #24
lmilcin wrote: post #13

One of my interview questions starts with "Can a program allocate more memory than is physically available on the server?" Everybody gets this wrong (which is funny for a binary question) but it starts an interesting discussion through which I hope to learn how much they know about OS and virtual memory.

Don't leave us hanging. The "obvious" answer would seem to be "yes" because of swap. But if everyone gets that wrong...

It's a vague question though. What kind of OS? What kind of runtime? Does "allocate" mean call malloc() and get non-null return value?

Re: In C, how do you know if the dynamic allocation succeeded?

#113

Post by matheusmoreira »

one_off_comment wrote: post #16
yjftsjthsd-h wrote: post #3

TLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request. I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).

There's gotta be some way to programmatically determine it, right? It may not be portable. It may require some system calls or something, but there's gotta be a way, right?

Maybe a system call that checks whether writing to a page would result in a fault? What if we had memory file descriptors for pages? We'd be able to use them with epoll and io_uring to asynchronously check whether it's safe to write to the pages they represent.

Re: In C, how do you know if the dynamic allocation succeeded?

#114

Post by kzrdude »

lmilcin wrote: post #37
yjftsjthsd-h wrote: post #33

Earlier quoted context omitted.

No, it's worse than that - the answer is "yes", because virtual memory + overcommit means that most of the time the OS will happily allow you to allocate more memory than physical+swap, and essentially gamble that you won't actually need all of it (and this is implemented because apparently that's almost always a winning bet).

Yeah. And the issue is that the actual problem happens sometime later when the application actually tries to use that memory. So you replaced an error that is relatively simple to handle with something that is impossible to handle reliably. So the operating system very much doesn't like to admit it doesn't have physical memory to back the area you are trying to use. Now it does not have a simple way to signal this to…

And we additionally end up in a feedback loop where coders don't check the return value from malloc(!) Since why bother when it never errors. Then we don't have enough resilience capital there either.

Re: In C, how do you know if the dynamic allocation succeeded?

#115

Post by Galanwe »

> If you use a system-level tool that reports the memory usage of your processes, you should look at the real memory usage.

Things are a bit more complicated than that. Because RSS will contain memory mapped to your process that could also be mapped by other processes. That is the sum of RSS on your machine is also higher than your physical memory.

That includes libraries dynamically linked to your executable, but more importantly shared memory mmapped to your process.

A more "fair" estimate exists in the form of PSS (or USS), that will list all mapped regions from all process, and account each process a proportional share of the region.

e.g. If 2 processes mmap `/dev/shm/foo` of 1GB, both will inherit 500GB by PSS computation.

Re: In C, how do you know if the dynamic allocation succeeded?

#116

Post by nextaccountic »

saagarjha wrote: post #75
joosters wrote: post #62

Earlier quoted context omitted.

It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on… the flexibility of a separate fork() then exec() means you can…

I mean, posix_spawn exists. It's a messy function, but its job is messy for exactly the reasons you describe. (FWIW, there are very few things you can legally perform between fork and exec.)

I think you mean between vfork and exec. The Linux manpage says:

Standard description

(From POSIX.1) The vfork() function has the same effect as fork(2), except that the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions.

Re: In C, how do you know if the dynamic allocation succeeded?

#117

Post by angry_octet »

trhway wrote: post #43
convolvatron wrote: post #17

Earlier quoted context omitted.

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

>there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now. Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people…

Virtual memory is always on... If you actually hit swap the system effectively deadlocks anyway because the swap daemon is too dumb by being too fair. Persistent memory might change this but it needs so much work.

Re: In C, how do you know if the dynamic allocation succeeded?

#118

Post by senderista »

Overcommit should not be necessary in order to just reserve a range of VM without committing it. Windows gets this right (requiring explicit commit to use memory if you explicitly reserved it earlier). Demand paging on Linux is super-convenient (and my designs exploit it heavily), but you can’t really write robust programs that way.

Re: In C, how do you know if the dynamic allocation succeeded?

#119

Post by SavantIdiot »

Try that on IAR EWARM on a Cortex-M0 target.

It will absolutely fail (first because size_t is only 32b).

What's behind malloc() is what matters.

That being said, I never would have expected that code to ever succeed! Shows how much I take memory allocation for granted on more sophisticated systems. I can't remember the last time I malloc'd more than a few megabytes.

Re: In C, how do you know if the dynamic allocation succeeded?

#120

Post by Rusky »

cturner wrote: post #36
alexgartrell wrote: post #31

Earlier quoted context omitted.

Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing…

"We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.

I'm not sure there's any causality between Windows preventing overcommit and Windows having slow process creation- there's nothing inherently slower about CreateProcess()/posix_spawn() than fork() + exec() as an API.

It seems more that overcommit is a workaround for the way fork() can potentially lead to copying the entire address space into new pages, but usually doesn't. Because CreateProcess() knows more precisely how much to allocate before it returns to the child process, it can just reserve that amount and signal an error immediately if there's not enough memory+swap to back it.

(And on the other hand, Windows has a lot of legacy and backwards compatibility behavior around processes that could easily explain the slower process creation independent of the API.)

Post reply on HN