Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vm.overcommit_memory=2 is the right setting for servers

ariadne.space

Post reply on HN
Print view View original

121–130 of 150 posts

Re: Vm.overcommit_memory=2 is the right setting for servers

#121

Post by self_awareness »

agilob wrote: post #110

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thro…

I'm wondering which overcommit strategy this example referrs to.

Because If my bitcoin price checker built on electron will start allocating all memory on the machine, then (assuming no overcommitting takes place) some arbitrary process (e.g. systemd) can get malloc error. But it's not systemd's fault the memory got eaten; so why it's being punished for low memory conditions?

It's like choosing a random person to be ejected from the plane.

Re: Vm.overcommit_memory=2 is the right setting for servers

#122

Post by bawolff »

c0l0 wrote: post #7

I realize this is mostly tangential to the article, but a word of warning for those who are about to mess with overcommit for the first time: In my experience, the extreme stance of "always do [thing] with overcommit" is just not defensible, because most (yes, also "server") software is just not written under the assumption that being able to deal with allocation failures in a meaningful way is a necessity. At best,…

> At best, there's an "malloc() or die"-like stanza in the source, and that's that.

In fairness, i don't know what else general purpose software is supposed to do here other than die. Its not like there is a graceful way to handle insufficient memory to run the program.

Re: Vm.overcommit_memory=2 is the right setting for servers

#123

Post by qhwudbebd »

johncolanduoni wrote: post #112
jclulow wrote: post #107

Earlier quoted context omitted.

> No non-embedded libc will actually return NULL This is just a Linux ecosystem thing. Other full size operating systems do memory accounting differently, and are able to correctly communicate when more memory is not available.

There are functions on many C allocators that are explicitly for non-trivial allocation scenarios, but what major operating system malloc implementation returns NULL? MSVC’s docs reserve the right to return NULL, but the actual code is not capable of doing so (because it would be a security nightmare).

I hack on various C projects on a linux/musl box, and I'm pretty sure I've seen musl's malloc() return 0, although possibly the only cases where I've triggered that fall into the 'unreasonably huge' category, where a typo made my enormous request fail some sanity check before even trying to allocate.

Re: Vm.overcommit_memory=2 is the right setting for servers

#124

Post by [deleted] »

agilob wrote: post #110

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thro…

[deleted]

Re: Vm.overcommit_memory=2 is the right setting for servers

#125

Post by jenadine »

johncolanduoni wrote: post #104
LordGrey wrote: post #53

Earlier quoted context omitted.

malloc() and friends may always return NULL. From the man page: If successful, calloc(), malloc(), realloc(), reallocf(), valloc(), and aligned_alloc() functions return a pointer to allocated memory. If there is an error, they return a NULL pointer and set errno to ENOMEM. In practice, I find a lot of code that does not check for NULL, which is rather distressing.

No non-embedded libc will actually return NULL. Very, very little practical C code actually relies only on specified behavior of the spec and will work with literally any compliant C compiler on any architecture, so I don’t find this particularly concerning. Usefully handling allocation errors is very hard to do well, since it infects literally every error handling path in your codebase. Any error handling that calls…

> No non-embedded libc will actually return NULL.

malloc(-1) should always return NULL. Malloc returns NULL if the virtual address space for a given process is exhausted.

It will not return NULL when the system is out of memory (depending on the overcommit settings)

Re: Vm.overcommit_memory=2 is the right setting for servers

#126

Post by fulafel »

sidewndr46 wrote: post #45
LordGrey wrote: post #3

For anyone not familiar with the meaning of '2' in this context: The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.…

Do any of the settings actually result in "malloc" or a similar function returning NULL?

Yes.

Re: Vm.overcommit_memory=2 is the right setting for servers

#127

Post by jenadine »

bawolff wrote: post #122
c0l0 wrote: post #7

I realize this is mostly tangential to the article, but a word of warning for those who are about to mess with overcommit for the first time: In my experience, the extreme stance of "always do [thing] with overcommit" is just not defensible, because most (yes, also "server") software is just not written under the assumption that being able to deal with allocation failures in a meaningful way is a necessity. At best,…

> At best, there's an "malloc() or die"-like stanza in the source, and that's that. In fairness, i don't know what else general purpose software is supposed to do here other than die. Its not like there is a graceful way to handle insufficient memory to run the program.

In theory, a process could just return an error for that specific operation, which would propagate to a "500 internal error" for this one request but not impact other operations. Could even take the hint to free some caches.

But in practice, I agree with you. This is just not worth it. So much work to handle it properly everywhere and it is really difficult to test every malloc failures.

So that's where an OOM killer might have a better strategy than just letting the last program that happen to allocate memory last to fail.

Re: Vm.overcommit_memory=2 is the right setting for servers

#128

Post by coryrc »

kentonv wrote: post #95

When your system is out of memory, you do not want to return an error to the next process that allocates memory. That might be an important process, it might have nothing to do with the reason the system is out of memory, and it might not be able to gracefully handle allocation failure (realistically, most programs can't). Instead, you want to kill the process that's hogging all the memory. The OOM killer heuristic i…

OOM killer often doesn't run soon enough for me; I've even left the machine for twenty minutes and it's still swapping hard.

And I do say "often" because it does sometimes work.

I have set all my Firefox processes near-maximum priority to kill for the OOM killer, but it didn't help.

Also don't forget about memory compression: only meaningful with overcommit.

Re: Vm.overcommit_memory=2 is the right setting for servers

#129

Post by rcxdude »

coryrc wrote: post #128
kentonv wrote: post #95

When your system is out of memory, you do not want to return an error to the next process that allocates memory. That might be an important process, it might have nothing to do with the reason the system is out of memory, and it might not be able to gracefully handle allocation failure (realistically, most programs can't). Instead, you want to kill the process that's hogging all the memory. The OOM killer heuristic i…

OOM killer often doesn't run soon enough for me; I've even left the machine for twenty minutes and it's still swapping hard. And I do say "often" because it does sometimes work. I have set all my Firefox processes near-maximum priority to kill for the OOM killer, but it didn't help. Also don't forget about memory compression: only meaningful with overcommit.

Yeah, you probably want something that activates a bit earlier. I think the issue is the OOM killer won't activate until essentially everything that can be paged out is paged out, and that includes most code pages, so the system enters a death spiral of paging code in and out and stops making progress towards a point where the OOM killer would kick in. There's userspace daemons like earlyoom that help a lot with this.
Post reply on HN