Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

41–50 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#41

Post by joosters »

secondcoming wrote: post #25
convolvatron wrote: post #17

Earlier quoted context omitted.

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left.

Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

Re: In C, how do you know if the dynamic allocation succeeded?

#42

Post by pjc50 »

convolvatron wrote: post #17
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed.

(I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)

Re: In C, how do you know if the dynamic allocation succeeded?

#43

Post by trhway »

convolvatron wrote: post #17
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

>there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now.

Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people not configuring virtual memory any more we get the situation you describe.

Re: In C, how do you know if the dynamic allocation succeeded?

#44

Post by convolvatron »

kibwen wrote: post #21
mirashii wrote: post #2

I'm a little disappointed that the article didn't answer the question, or at least try to. A discussion of using read/write vs mincore vs trying to catch a SIGSEGV would've been a nice addition.

The answer is that you kind of can't. You're at the mercy of the OS to give you accurate information, and malloc as an interface isn't set up to distinguish between virtual memory and "actual" memory. We could imagine a separate interface that would allow the OS to communicate this distinction (or hacks like you allude to), but I don't know of any standard approach.

before linux - every unix OS would fail a page allocation if there wasn't a backing store. full stop.

this worked really well

Re: In C, how do you know if the dynamic allocation succeeded?

#45

Post by lmilcin »

underdeserver wrote: post #39
lmilcin wrote: post #12

Earlier quoted context omitted.

Not the same thing. malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.

TIL, unless you explicitly disable memory overcommit, it can and will overcommit. This is crazy to me.

It is not. It is actually very useful and advantageous for many reasons.

The disadvantage is the pretty shitty failure mode.

But in practice, I have never seen an operating system that was not a toy that was able to reliably, gracefully handle out of memory condition.

Re: In C, how do you know if the dynamic allocation succeeded?

#46

Post by pjc50 »

cturner wrote: post #36
alexgartrell wrote: post #31

Earlier quoted context omitted.

Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing…

"We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.

Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975

For ages cygwin had to emulate it manually by hand-copying the process.

Re: In C, how do you know if the dynamic allocation succeeded?

#47

Post by dekhn »

pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

The biggest reason overcommit exists is because it allows the system to operate more efficiently. The reality is most applications touch only some of the pages they allocate, and it's silly for the system to fail a malloc. Often times other expensive cleanup activities can be deferred (you don't really want to drop a handy directory entry cache just so an app can be sure it got physically backed memory for its request, 99.99% of the time).

IIUC Linux was really the first OS to make overcommit so prominent. Most systems were a lot more conservative.

Re: In C, how do you know if the dynamic allocation succeeded?

#48

Post by dekhn »

underdeserver wrote: post #39
lmilcin wrote: post #12

Earlier quoted context omitted.

Not the same thing. malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.

TIL, unless you explicitly disable memory overcommit, it can and will overcommit. This is crazy to me.

So, uh, how do you feel about fractional reserve banking? Nearly all banks worldwide practice it. Statistically, it's not impossible that the entire world financial system could collapse due to uncorrelated bank runs.

Re: In C, how do you know if the dynamic allocation succeeded?

#49

Post by pjc50 »

dekhn wrote: post #6

I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

Swap space is the Federal Reserve of memory allocation.

Re: In C, how do you know if the dynamic allocation succeeded?

#50

Post by joosters »

jimbob45 wrote: post #28
andi999 wrote: post #7

This has nothing to do with C. The problem/feature is with the systemcall that does the allocation, and any language has to use it.

Is your issue that the system call doesn't return enough diagnostic information? If so, how would you have done it differently? I'm asking out of curiosity, not out of a reflexive instinct to defend C (which has many problems).

The difficulty is that the lack of memory might be discovered days after it was allocated (an over-committing allocator simply doesn’t know at the time whether there will be memory or not) - how do you asynchronously tell the program that this allocation has now failed in a high-level way?

Generally, UNIX will let you know about the missing memory by sending the process a signal. But by that point, there’s not much that can be done to fix things up - and remember, all the fix up code would have to run without allocating any more memory itself. That’s extremely tricky in C, and nigh-on impossible in other languages.

Post reply on HN