Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

181–190 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#181

Post by 01100011 »

willbudd wrote: post #106
shoo wrote: post #94

Earlier quoted context omitted.

I first learned about this while trying to understand processes being killed by OOM in production. We had python 2.x batch jobs being executed by long-running python worker processes -- some of the arbitrary application code in some of the arbitrary batch jobs would occasionally want to execute some command line tool in a new process, and to create the new process under the hood python's subprocess library would fork…

Sounds to me like you (or Python devs?) should be looking at clone(), which is basically a lower level version of fork() that provides precise control over every detail with which the new process is created, including what (if any) parent memory you want to share. The manual page is quite informative: man clone

This. When my team hit the issue we just made a subprocess replacement which relied on clone instead. Before that, we would trigger OOM when trying to run external commands while still having 30% free memory.

Re: In C, how do you know if the dynamic allocation succeeded?

#182

Post by varispeed »

dahart wrote: post #124
Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

But this feature is also why malloc on Windows can take a long time to return - seconds or more if you malloc too much. I recently bumped into this problem on the GPU. Since GPU memory in the Windows Display Driver Model requires it to be backed by the host machine’s virtual memory, a malloc on the GPU when running low on space can end up searching & even defragging virtual memory to satisfy the request. Crazy! Just…

I have a faint memory of writing custom memory manager and allocating memory in bulk upon program start. But it was still a mess. Eventually had to periodically pause and defragment the memory allocations, rewriting pointers and so on. Well, at least it was fast.

Re: In C, how do you know if the dynamic allocation succeeded?

#183

Post by pjc50 »

magicalhippo wrote: post #84
pjc50 wrote: post #42

Earlier quoted context omitted.

But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)

> But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy th…

> Of course, kinda hard to fix this now..

Well, yes, if you can break backwards compatibility you can do anything. Except run all the existing software.

Re: In C, how do you know if the dynamic allocation succeeded?

#184

Post by gnubison »

bitwize wrote: post #99
joosters wrote: post #41

Earlier quoted context omitted.

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

That's just the thing. fork(2), like much of POSIX, was a mistake. The proper semantics for starting a new process is something like posix_spawn or Win32 CreateProcess, i.e., you specify an executable image to start.

It’s not like you can blame POSIX for this — they had to standardize existing behavior.

Re: In C, how do you know if the dynamic allocation succeeded?

#185

Post by andi999 »

jimbob45 wrote: post #28
andi999 wrote: post #7

This has nothing to do with C. The problem/feature is with the systemcall that does the allocation, and any language has to use it.

Is your issue that the system call doesn't return enough diagnostic information? If so, how would you have done it differently? I'm asking out of curiosity, not out of a reflexive instinct to defend C (which has many problems).

My first issue is with the article, while it seems to say it is about C, it is about a different problem which is illustrated with a C program. So what I would do differently (now that memory is cheap) : do not do overcommiting, so the OOM is out of the window. Do a different approach to fork, so you can exec commands and don't need to allocate the world. Actually just using the ms windows memory model would be fine.

Re: In C, how do you know if the dynamic allocation succeeded?

#188

Post by foobarian »

lmilcin wrote: post #176
foobarian wrote: post #112

Earlier quoted context omitted.

It's a vague question though. What kind of OS? What kind of runtime? Does "allocate" mean call malloc() and get non-null return value?

If you can point all that stuff out I will already be a little happier as an interviewer. Most people just say "no".

Fair enough, I wouldn’t be happy with a single word either, no matter if “yes” or “no.”

Re: In C, how do you know if the dynamic allocation succeeded?

#189

Post by _3u10 »

dekhn wrote: post #92
_3u10 wrote: post #71

Earlier quoted context omitted.

It is impossible. Only a moron of magnificent magnitude would fail to print additional cash to cover the run. The problems caused by the feds failure to lend to First Bank of America during the Great Depression are well understood by the central banks. What would likely happen is the overnight rate would go up to 12%, and additional money would be printed to cover withdrawals for the month or two most people would be…

i mean, the world financial system did almost just collapse, not that long ago, https://en.wikipedia.org/wiki/Financial_crisis_of_2007%E2%80... more or less due to the confluence of several adverse events, each of which probably could have been buffered on its own. When you say additional money would be printed, I assume you mean the M0 money supply would be increased?

In the case of a bank run specifically yes. In the more general case of a deleveraging (2008) it would be QE / asset purchases.
Post reply on HN