Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

61–70 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#61

Post by gpderetta »

giomasce wrote: post #59
joosters wrote: post #41

Earlier quoted context omitted.

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv.

Vfork is usually a better solution.

Edit: and yes, I would love to be able to disable overcommit per process.

Re: In C, how do you know if the dynamic allocation succeeded?

#62

Post by joosters »

giomasce wrote: post #59
joosters wrote: post #41

Earlier quoted context omitted.

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

It’s a weakness of the fork()+exec() model, for sure. However, creating a fork_and_execve() API is extremely tricky. Just think of all the innumerable setup options you would need to give it, e.g. what file handles should be closed or left open? What directory should it start in? What environment variables should be set - or cleared? And on and on and on…

the flexibility of a separate fork() then exec() means you can set up the initial state of a new process exactly as you want, by doing whatever work is needed between the two calls. If you merge them into one, then you will never be able to encapsulate all of that.

Re: In C, how do you know if the dynamic allocation succeeded?

#63

Post by dekhn »

joosters wrote: post #60
dekhn wrote: post #56

Earlier quoted context omitted.

no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-... I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's docume…

Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other. (I’m talking in general here - not Linux specifically)

dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.

Re: In C, how do you know if the dynamic allocation succeeded?

#64

Post by OldHand2018 »

gpderetta wrote: post #61
giomasce wrote: post #59

Earlier quoted context omitted.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier). Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

They do. Man posix_spawn. It comes with its own dsl to be able to support a small subset of all operations that are often performed between fork and execv. Vfork is usually a better solution. Edit: and yes, I would love to be able to disable overcommit per process.

This paper [1] argues that fork needs to be deprecated in favor of posix_spawn() or other more modern solutions. It claims that, among many other reasons, that fork encourages overcommit and that programs such as Redis are extraordinarily constrained if overcommit is disabled - because of fork.

[1] https://dl.acm.org/doi/10.1145/3317550.3321435

Re: In C, how do you know if the dynamic allocation succeeded?

#65

Post by joosters »

dekhn wrote: post #63
joosters wrote: post #60

Earlier quoted context omitted.

Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other. (I’m talking in general here - not Linux specifically)

dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.

I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?

Re: In C, how do you know if the dynamic allocation succeeded?

#67

Post by dekhn »

joosters wrote: post #65
dekhn wrote: post #63

Earlier quoted context omitted.

dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.

I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?

if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk instead of RAM).

Everything I'm describing is about a busy server with heterogenous workloads of specific types.

Re: In C, how do you know if the dynamic allocation succeeded?

#68

Post by _3u10 »

wahern wrote: post #40
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.

Even without overcommit swap makes the system work better as unused pages can be written to disk and that memory used for disk cache and I think can also help with defragmentation of memory not sure how that process actually works.

Re: In C, how do you know if the dynamic allocation succeeded?

#69

Post by pcwalton »

convolvatron wrote: post #17
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

> wouldn't you just account the COW pages against the parent until they are copied?

That's what Linux does. But how do you return ENOMEM when the copy does happen and now the system is out of memory? Memory writes don't return error codes. The best you could do is send a signal, which is exactly what the OOM killer does.

Re: In C, how do you know if the dynamic allocation succeeded?

#70

Post by q-big »

dekhn wrote: post #6

I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

> I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

What if you are worried about both? ;-)

Post reply on HN