Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vm.overcommit_memory=2 is the right setting for servers

ariadne.space

Post reply on HN
Print view View original

61–70 of 150 posts

Re: Vm.overcommit_memory=2 is the right setting for servers

#61

Post by loeg »

toast0 wrote: post #55
ycombinatrix wrote: post #19

Earlier quoted context omitted.

How does fork() work with vm.overcommit=2? A forked process would assume memory is already allocated, but I guess it would fail when writing to it as if vm.overcommit is set to 0 or 1.

As pm215 states, it doubles your memory commit. It's somewhat common for large programs/runtimes that may fork at runtime to spawn an intermediary process during startup to use for runtime forks to avoid the cost of CoW on memory and mapppings and etc where the CoW isn't needed or desirable; but redis has to fork the actual service process because it uses CoW to effectively snapshot memory.

It seems like a wrong accounting to count CoWed pages twice.

Re: Vm.overcommit_memory=2 is the right setting for servers

#62

Post by loeg »

jcalvinowens wrote: post #30
loeg wrote: post #27

Earlier quoted context omitted.

If the overcommit ratio is 1, there is no portion rendered unusable? This seems to contradict your "necessarily" wastes RAM claim?

Read the comment again, that wasn't the only one I mentioned.

Please point out what you're talking about, because the comment is short and I read it fully multiple times now.

Re: Vm.overcommit_memory=2 is the right setting for servers

#63

Post by loeg »

mahkoh wrote: post #38
loeg wrote: post #25

Earlier quoted context omitted.

Why? Most COWed pages will remain untouched. They only need to allocate when touched.

The point of disabling overcommit, as per the article, is that all pages in virtual memory must be backed by physical memory at all times. Therefore all virtual memory must reserve physical memory at the time of the fork call, even if the contents of the pages only get copied when they are touched.

Surely e.g. shared memory segments that are mapped by multiple processes are not double-counted? So it's only COW memory in particular that gets this treatment? Linux could just not do that.

Re: Vm.overcommit_memory=2 is the right setting for servers

#64

Post by kibwen »

loeg wrote: post #63
mahkoh wrote: post #38

Earlier quoted context omitted.

The point of disabling overcommit, as per the article, is that all pages in virtual memory must be backed by physical memory at all times. Therefore all virtual memory must reserve physical memory at the time of the fork call, even if the contents of the pages only get copied when they are touched.

Surely e.g. shared memory segments that are mapped by multiple processes are not double-counted? So it's only COW memory in particular that gets this treatment? Linux could just not do that.

But forking duplicates the process space and has to assume that a write might happen, so it has to defensively reserve enough for the new process if overcommit is off.

Re: Vm.overcommit_memory=2 is the right setting for servers

#65

Post by kibwen »

loeg wrote: post #61
toast0 wrote: post #55

Earlier quoted context omitted.

As pm215 states, it doubles your memory commit. It's somewhat common for large programs/runtimes that may fork at runtime to spawn an intermediary process during startup to use for runtime forks to avoid the cost of CoW on memory and mapppings and etc where the CoW isn't needed or desirable; but redis has to fork the actual service process because it uses CoW to effectively snapshot memory.

It seems like a wrong accounting to count CoWed pages twice.

Not if your goal is to make it such that OOM can only occur during allocation failure, and not during an arbitrary later write, as the OP purports to want.

Re: Vm.overcommit_memory=2 is the right setting for servers

#66

Post by toast0 »

loeg wrote: post #61
toast0 wrote: post #55

Earlier quoted context omitted.

As pm215 states, it doubles your memory commit. It's somewhat common for large programs/runtimes that may fork at runtime to spawn an intermediary process during startup to use for runtime forks to avoid the cost of CoW on memory and mapppings and etc where the CoW isn't needed or desirable; but redis has to fork the actual service process because it uses CoW to effectively snapshot memory.

It seems like a wrong accounting to count CoWed pages twice.

It's not really wrong. For something like redis, you could potentially fork and the child gets stuck for a long time and in the meantime the whole cache in the parent is rewritten. In that case, even though the cache is fixed size / no new allocations, all of the pages are touched and so the total used memory is double from before the fork. If you want to guarantee allocation failures rather than demand paging failures, and you don't have enough ram/swap to back twice the allocations, you must fail the fork.

On the other hand, if you have a pretty good idea that the child will finish persisting and exit before the cache is fully rewritten, double is too much. There's not really a mechanism for that though. Even if you could set an optimistic multiplier for multiple mapped CoW pages, you're back to demand paging failures --- although maybe it's still worthwhile.

Re: Vm.overcommit_memory=2 is the right setting for servers

#67

Post by ece »

Tuna-Fish wrote: post #31
kg wrote: post #10

Earlier quoted context omitted.

I run my development VM with overcommit disabled and the way stuff fails when it runs out of memory is really confusing and mysterious sometimes. It's useful for flushing out issues that would otherwise cause system degradation w/overcommit enabled, so I keep it that way, but yeah... doing it in production with a bunch of different applications running is probably asking for trouble.

The fundamental problem is that your machine is running software from a thousand different projects or libraries just to provide the basic system, and most of them do not handle allocation failure gracefully. If program A allocates too much memory and overcommit is off, that doesn't necessarily mean that A gets an allocation failure. It might also mean that code in library B in background process C gets the failure,…

This is a better explanation and fix than others I've seen. There will be differences between desktop and server uses, but misbehaving applications and libraries exist on both.

Re: Vm.overcommit_memory=2 is the right setting for servers

#68

Post by laurencerowe »

silon42 wrote: post #56
laurencerowe wrote: post #43

Disabling overcommit on V8 servers like Deno will be incredibly inefficient. Your process might only need ~100MB of memory or so but V8's cppgc caged heap requires a 64GB allocation in order to get a 32GB aligned area in which to contain its pointers. This is a security measure to prevent any possibility of out of cage access.

Maybe it should use MAP_NORESERVE ?

I expect it does already, but I don’t think it would help here:

> In mode 2 the MAP_NORESERVE flag is ignored.

https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

Re: Vm.overcommit_memory=2 is the right setting for servers

#69

Post by kg »

vin10 wrote: post #23
kg wrote: post #10

Earlier quoted context omitted.

I run my development VM with overcommit disabled and the way stuff fails when it runs out of memory is really confusing and mysterious sometimes. It's useful for flushing out issues that would otherwise cause system degradation w/overcommit enabled, so I keep it that way, but yeah... doing it in production with a bunch of different applications running is probably asking for trouble.

> he way stuff fails when it runs out of memory is really confusing have you checked what your `vm.overcommit_ratio` is? If its curious what kind of failures you are alluding to.

The main scenario that caused me a lot of grief is temporary RAM usage spikes, like a single process run during a build that uses ~8gb of RAM or more for a mere few seconds and then exits. In some cases the oom killer was reaping the wrong process or the build was just failing cryptically and if I examined stuff like top I wouldn't see any issue, plenty of free RAM. The tooling for examining this historical memory usage is pretty bad, my only option was to look at the oom killer logs and hope that eventually the culprit would show up.

Thanks for the tip about vm.overcommit_ratio though, I think it's set to the default.

Re: Vm.overcommit_memory=2 is the right setting for servers

#70

Post by godelski »

dbdr wrote: post #20
LordGrey wrote: post #3

For anyone not familiar with the meaning of '2' in this context: The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.…

> exceed swap + a configurable amount (default is 50%) of physical RAM Naive question: why is this default 50%, and more generally why is this not the entire RAM, what happens to the rest?

There's a lot of options. If you want to go down the rabbithole try typing `sysctl -a | grep -E "^vm"` and that'll give you a lot of things to google ;)
Post reply on HN