Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

81–90 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#81

Post by wongarsu »

dekhn wrote: post #67
joosters wrote: post #65

Earlier quoted context omitted.

I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?

if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk ins…

I think what joosters is trying to say that instead of overcommitting a system could just allocate memory directly in swap. Both creating the page in swap and swapping it in on first write can be mostly optimized away apart from the bookkeeping, but you guarantee that you always have enough RAM+Swap to provide what you have committed to.

Re: In C, how do you know if the dynamic allocation succeeded?

#83

Post by scottlamb »

gpderetta wrote: post #61
giomasce wrote: post #59

Earlier quoted context omitted.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.

> Vfork is usually a better solution.

I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions."

So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In practice, I believe it's fine, and on glibc posix_spawn is apparently written in terms of vfork, but libc is allowed to make assumptions that portable programs shouldn't.

Story time: I once worked on a Unix-based system with no MMU. The fork implementation did something insane: it looked for things that were possibly pointers (four-byte aligned memory locations which can be interpreted as valid physical memory locations on this system allocated to that program) and adjusted them. It was kind of like a conservative GC in that it treated anything that looked like a pointer as if it were a pointer. But no reasonable person would call modifying memory that may or may not be pointers to be "conservative". Amazingly, it worked most of the time, but sometimes strings got corrupted, and as a workaround there were a bunch of places where string buffers were fully zeroed where just a NUL byte would otherwise do.

This was early in my career, and I didn't design the system anyway. If I were working on it today I'd remove the fork implementation and make everything use vfork and/or posix_spawn instead. Apparently that's what posix_spawn was made for.

Another weird problem with the lack of MMU: the compiler also didn't use register-based addressing (what's the term? like position-independent code but for the data segment?), so global variables were truly global, not just global to the process. A bunch of code needed to be "deglobalized" to deal with this. I wanted to improve the compiler, but long story short I got offered another job first.

Re: In C, how do you know if the dynamic allocation succeeded?

#84

Post by magicalhippo »

pjc50 wrote: post #42
convolvatron wrote: post #17

Earlier quoted context omitted.

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)

> But what happens when the kernel needs to copy one and has run out of memory?

Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process.

If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages.

Of course, kinda hard to fix this now...

Re: In C, how do you know if the dynamic allocation succeeded?

#85

Post by zajio1am »

IMO strict memory accounting misses the point. It can ensure that all allocated pages fits to VM (or return error during allocation), but more pragmatic memory constraint is whether working set of pages (including code/mmap-based pages) fits in physical RAM. If that is not satisfied, system/application crawls to a halt due to page thrashing and some kind of OOM killer is needed. And that may happen even if strict memory accounting is satisfied.

Re: In C, how do you know if the dynamic allocation succeeded?

#86

Post by foota »

pjc50 wrote: post #49
dekhn wrote: post #6

I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

Swap space is the Federal Reserve of memory allocation.

Conspiracy theorists hate it?

Re: In C, how do you know if the dynamic allocation succeeded?

#87

Post by pcwalton »

scottlamb wrote: post #83
gpderetta wrote: post #61

Earlier quoted context omitted.

They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.

> Vfork is usually a better solution. I think the biggest downside to vfork is (from the Linux manpage) "the behavior is undefined if the process created by vfork() [...] calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions." So strictly speaking I think doing any of those operations documented by posix_spawn yourself between vfork and execv is undefined behavior. In…

No-MMU Linux usually just disables fork and uses vfork instead. The really annoying part is that you can't use shared libraries without doubling the size of function pointers (FD-PIC ABI).

Re: In C, how do you know if the dynamic allocation succeeded?

#88

Post by milkey_mouse »

Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

I've found overcommit useful for some scientific computing applications, but you're right it's not the most intuitive default. You can disable memory overcommit on Linux for the same behavior as Windows, if you want.

https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

Also, I found this post that suggests Windows technically does overcommit memory, but only for stacks(‽): https://superuser.com/questions/1194263/will-microsoft-windo...

Re: In C, how do you know if the dynamic allocation succeeded?

#89

Post by tialaramex »

Const-me wrote: post #82

That’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.

If you really want that you can just tell Linux not to overcommit. I can foresee your next question though, "Wait, why did I run out of memory? Stupid Linux, I have plenty of RAM".

Re: In C, how do you know if the dynamic allocation succeeded?

#90

Post by wmf »

joosters wrote: post #62
giomasce wrote: post #59

Earlier quoted context omitted.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on… the flexibility of a separate fork() then exec() means you can…

A third option would be the ability to create a blank process, configure it, then exec within it. I think Mach allows this.
Post reply on HN