Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In C, how do you know if the dynamic allocation succeeded?

lemire.me

Post reply on HN
Print view View original

51–60 of 198 posts

Re: In C, how do you know if the dynamic allocation succeeded?

#51

Post by pjc50 »

wahern wrote: post #40
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.

How does Solaris handle this? Or Darwin?

Re: In C, how do you know if the dynamic allocation succeeded?

#52

Post by OldHand2018 »

yjftsjthsd-h wrote: post #33
floxy wrote: post #24

Earlier quoted context omitted.

Don't leave us hanging. The "obvious" answer would seem to be "yes" because of swap. But if everyone gets that wrong...

No, it's worse than that - the answer is "yes", because virtual memory + overcommit means that most of the time the OS will happily allow you to allocate more memory than physical+swap, and essentially gamble that you won't actually need all of it (and this is implemented because apparently that's almost always a winning bet).

Processes come and go!

The OS doesn't have to gamble that you won't actually need all the memory you allocate, it could just be a gamble that another memory hogging process exits before you need to use all of your memory, or that you don't need to use all of your memory at the same time.

Re: In C, how do you know if the dynamic allocation succeeded?

#53

Post by [deleted] »

pjc50 wrote: post #49
dekhn wrote: post #6

I once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.

Swap space is the Federal Reserve of memory allocation.

[deleted]

Re: In C, how do you know if the dynamic allocation succeeded?

#54

Post by joosters »

dekhn wrote: post #47
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

The biggest reason overcommit exists is because it allows the system to operate more efficiently. The reality is most applications touch only some of the pages they allocate, and it's silly for the system to fail a malloc. Often times other expensive cleanup activities can be deferred (you don't really want to drop a handy directory entry cache just so an app can be sure it got physically backed memory for its reques…

In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap

Re: In C, how do you know if the dynamic allocation succeeded?

#55

Post by joosters »

wahern wrote: post #40
pcwalton wrote: post #15

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being…

> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.

Solaris does have this problem! If you have a huge program running, fork()img it can fail on Solaris, even if you only want to just exec a tiny program. The key way of avoiding this is to ensure you have lots and lots of swap space.

Re: In C, how do you know if the dynamic allocation succeeded?

#56

Post by dekhn »

joosters wrote: post #54
dekhn wrote: post #47

Earlier quoted context omitted.

The biggest reason overcommit exists is because it allows the system to operate more efficiently. The reality is most applications touch only some of the pages they allocate, and it's silly for the system to fail a malloc. Often times other expensive cleanup activities can be deferred (you don't really want to drop a handy directory entry cache just so an app can be sure it got physically backed memory for its reques…

In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap

no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-...

I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.

Re: In C, how do you know if the dynamic allocation succeeded?

#57

Post by OldHand2018 »

secondcoming wrote: post #25
convolvatron wrote: post #17

Earlier quoted context omitted.

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks…

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.

Let's say you malloc some memory and the computer actually has everything available.

Everything is great, up until some other process on your system does a fork bomb of an infinitely recursive program that allocates nothing on the heap. You've just got a whole lot of quickly growing stacks hoovering up your physical memory pages.

Re: In C, how do you know if the dynamic allocation succeeded?

#58

Post by wahern »

pjc50 wrote: post #51
wahern wrote: post #40

Earlier quoted context omitted.

> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.

How does Solaris handle this? Or Darwin?

Solaris has strict memory accounting; fork will fail if the system can't guarantee space for all non-shared anonymous memory. macOS has overcommit (all BSDs do to some extent, at least for fork), but it also automatically creates swap space so you rarely encounter issues one way or another in practice.

fork and malloc can also fail in Linux even with overcommit enabled (rlimits, but also OOM killer racing with I/O page dirtying triggering best-effort timeout), so Linux buys you a little convenience at the cost of making it impossibly difficult to actually guarantee behavior when it matters most.

Re: In C, how do you know if the dynamic allocation succeeded?

#59

Post by giomasce »

joosters wrote: post #41
secondcoming wrote: post #25

Earlier quoted context omitted.

Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.

Please don’t blindly declare it ‘totally dumb’. If you disallow overcommit, you can end up with a system that can’t fork and run /bin/true, even if there are gigabytes of memory left. Both styles of memory allocation have their uses, and their drawbacks, but please understand them before declaring many OS designers as stupid and dumb.

I suppose vfork might help with that (though I don't understand why they don't directly add fork_and_execve, it would seem much easier).

Also, the problem with Linux is not having overcommit, but notv being able to choose when to overcommit and when not. Windows makes that easier, AFAIU.

Re: In C, how do you know if the dynamic allocation succeeded?

#60

Post by joosters »

dekhn wrote: post #56
joosters wrote: post #54

Earlier quoted context omitted.

In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap

no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-... I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's docume…

Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other.

(I’m talking in general here - not Linux specifically)

Post reply on HN