Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vm.overcommit_memory=2 is the right setting for servers

ariadne.space

Post reply on HN
Print view View original

31–40 of 150 posts

Re: Vm.overcommit_memory=2 is the right setting for servers

#31

Post by Tuna-Fish »

kg wrote: post #10
c0l0 wrote: post #7

I realize this is mostly tangential to the article, but a word of warning for those who are about to mess with overcommit for the first time: In my experience, the extreme stance of "always do [thing] with overcommit" is just not defensible, because most (yes, also "server") software is just not written under the assumption that being able to deal with allocation failures in a meaningful way is a necessity. At best,…

I run my development VM with overcommit disabled and the way stuff fails when it runs out of memory is really confusing and mysterious sometimes. It's useful for flushing out issues that would otherwise cause system degradation w/overcommit enabled, so I keep it that way, but yeah... doing it in production with a bunch of different applications running is probably asking for trouble.

The fundamental problem is that your machine is running software from a thousand different projects or libraries just to provide the basic system, and most of them do not handle allocation failure gracefully. If program A allocates too much memory and overcommit is off, that doesn't necessarily mean that A gets an allocation failure. It might also mean that code in library B in background process C gets the failure, and fails in a way that puts the system in a state that's not easily recoverable, and is possibly very different every time it happens.

For cleanly surfacing errors, overcommit=2 is a bad choice. For most servers, it's much better to leave overcommit on, but make the OOM killer always target your primary service/container, using oom-score-adj, and/or memory.oom.group to take out the whole cgroup. This way, you get to cleanly combine your OOM condition handling with the general failure case and can restart everything from a known foundation, instead of trying to soldier on while possibly lacking some piece of support infrastructure that is necessary but usually invisible.

Re: Vm.overcommit_memory=2 is the right setting for servers

#33

Post by pizlonator »

This is such an old debate. The real answer, as with all such things, is "it depends".

Two reasons why overcommit is a good idea:

- It lets you reserve memory and use the dirtying of that memory to be the thing that commits it. Some algorithms and data structures rely on this strongly (i.e. you would have to use a significantly different algorithm, which is demonstrably slower or more memory intensive, if you couldn't rely on overcommit).

- Many applications have no story for out-of-memory other halting. You can scream and yell at them to do better, but that won't help, because those apps that find themselves in that supposedly-bad situation ended up there for complex and well-considered reasons. My favorite: having complex OOM error handling paths is the worst kind of attack surface, since it's hard to get test coverage for it. So, it's better to just have the program killed instead, because that nixes the untested code path. For those programs, there's zero value in having the memory allocator be able to report OOM conditions other than by asserting in prod that mmap/madvise always succeed, which then means that the value of not overcommitting is much smaller.

Are there server apps where the value of gracefully handling out of memory errors outweighs the perf benefits of overcommit and the attack surface mitigation of halting on OOM? Yeah! But I bet that not all server apps fall into that bucket

Re: Vm.overcommit_memory=2 is the right setting for servers

#34

Post by Tuna-Fish »

silon42 wrote: post #14
Animats wrote: post #6

Setting 2 is still pretty generous. It means "Kernel does not allow allocations that exceed swap + (RAM × overcommit_ratio / 100)." It's not a "never swap or overcommit" setting. You can still get into thrashing by memory overload. We may be entering an era when everyone in computing has to get serious about resource consumption. NVidia says GPUs are going to get more expensive for the next five years. DRAM prices ar…

For me, on the desktop, thrashing overload is the most common way the Linux system effectively crashes... (I've left it overnight a few times, sometimes it recovered, but not always). I'm not disabling overcommit for now, but maybe I should.

disabling overcommit does not fix trashing. Reducing the size of your swap does.

Re: Vm.overcommit_memory=2 is the right setting for servers

#35

Post by pm215 »

loeg wrote: post #25
wmf wrote: post #4

This doesn't address the fact that forking large processes requires either overcommit or a lot of swap. That may be the source of the Redis problem.

Why? Most COWed pages will remain untouched. They only need to allocate when touched.

Because the point of forbidding overcommit is to ensure that the only time you can discover you're out of memory is when you make a syscall that tries (explicitly or implicitly) to allocate more memory. If you don't account the COW pages to both the parent and the child process, you have a situation where you can discover the out of memory condition when the process tries to dirty the RAM and there's no page available to do that with...

Re: Vm.overcommit_memory=2 is the right setting for servers

#36

Post by jcalvinowens »

loeg wrote: post #29
jcalvinowens wrote: post #28

Earlier quoted context omitted.

Because userspace rarely actually faults in all the pages it allocates.

Surely the source of the waste here is the userspace program not using the memory it allocated, rather than whether or not the kernel overcommits memory. Attributing this to overcommit behavior is invalid.

Obviously. But all programs do that and have done it forever, it's literally the very reason overcommit exists.

Re: Vm.overcommit_memory=2 is the right setting for servers

#37

Post by Tuna-Fish »

loeg wrote: post #25
wmf wrote: post #4

This doesn't address the fact that forking large processes requires either overcommit or a lot of swap. That may be the source of the Redis problem.

Why? Most COWed pages will remain untouched. They only need to allocate when touched.

If you have overcommit on, that happens. But if you have it off, it has to assume the worst case, otherwise there can be a failure when someone writes to a page.

Re: Vm.overcommit_memory=2 is the right setting for servers

#38

Post by mahkoh »

loeg wrote: post #25
wmf wrote: post #4

This doesn't address the fact that forking large processes requires either overcommit or a lot of swap. That may be the source of the Redis problem.

Why? Most COWed pages will remain untouched. They only need to allocate when touched.

The point of disabling overcommit, as per the article, is that all pages in virtual memory must be backed by physical memory at all times. Therefore all virtual memory must reserve physical memory at the time of the fork call, even if the contents of the pages only get copied when they are touched.

Re: Vm.overcommit_memory=2 is the right setting for servers

#40

Post by pm215 »

ycombinatrix wrote: post #19
blibble wrote: post #8

redis uses the copy-on-write property of fork() to implement saving which is elegant and completely legitimate

How does fork() work with vm.overcommit=2? A forked process would assume memory is already allocated, but I guess it would fail when writing to it as if vm.overcommit is set to 0 or 1.

I believe (per the stuff at the bottom of https://www.kernel.org/doc/Documentation/vm/overcommit-accou... ) that the kernel does the accounting of how much memory the new child process needs and will fail the fork() if there isn't enough. All the COW pages should be in the "shared anonymous" category so get counted once per user (i.e. once for the parent process, once for the child), ensuring that the COW copy can't fail if the fork succeeded.
Post reply on HN