Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vm.overcommit_memory=2 is the right setting for servers

ariadne.space

Post reply on HN
Print view View original

91–100 of 150 posts

Re: Vm.overcommit_memory=2 is the right setting for servers

#91

Post by Wowfunhappy »

crote wrote: post #47
dbdr wrote: post #20

Earlier quoted context omitted.

> exceed swap + a configurable amount (default is 50%) of physical RAM Naive question: why is this default 50%, and more generally why is this not the entire RAM, what happens to the rest?

Just a guess, but I reckon it doesn't account for things like kernel memory usage, such as caches and buffers. Assigning 100% of physical RAM to applications is probably going to have a Really Bad Outcome.

But the memory being used by the kernel has already been allocated by the kernel. So obviously that RAM isn't available.

I can understand leaving some amount free in case the kernel needs to allocate additional memory in the future, but anything near half seems like a lot!

Re: Vm.overcommit_memory=2 is the right setting for servers

#92

Post by inkyoto »

pm215 wrote: post #35
loeg wrote: post #25

Earlier quoted context omitted.

Why? Most COWed pages will remain untouched. They only need to allocate when touched.

Because the point of forbidding overcommit is to ensure that the only time you can discover you're out of memory is when you make a syscall that tries (explicitly or implicitly) to allocate more memory. If you don't account the COW pages to both the parent and the child process, you have a situation where you can discover the out of memory condition when the process tries to dirty the RAM and there's no page availabl…

The described scenario (and, consequently, a concern) is mostly a philosophical question or a real concern for a very specific workload.

Memory allocation is a highly non-deterministic process which highly depends on the code path, and it is generally impossible to predict how the child will handle its own memory space – it can be little or it can be more (relatively to the parent), and it is usually somewhere in the middle. Most daemons, for example, consuming next to zero extra memory after forking.

The Ruby garbage collector «mark-and-sweep» (old versions of Ruby – 1.8 and 1.9) and Python reference counting (the Instagram case) bugs are prime examples of pathological cases when a child would walk over its data pages, making each dirty and causing a system collapse, but the bugs have been fixed or workarounds have been applied. Honourable mention goes to Redis in a situation when THP (transparent huge pages) are enabled.

No heuristics exist out there that would turn memory allocation into a deterministic process.

Re: Vm.overcommit_memory=2 is the right setting for servers

#93

Post by MrDrMcCoy »

Tuna-Fish wrote: post #31
kg wrote: post #10

Earlier quoted context omitted.

I run my development VM with overcommit disabled and the way stuff fails when it runs out of memory is really confusing and mysterious sometimes. It's useful for flushing out issues that would otherwise cause system degradation w/overcommit enabled, so I keep it that way, but yeah... doing it in production with a bunch of different applications running is probably asking for trouble.

The fundamental problem is that your machine is running software from a thousand different projects or libraries just to provide the basic system, and most of them do not handle allocation failure gracefully. If program A allocates too much memory and overcommit is off, that doesn't necessarily mean that A gets an allocation failure. It might also mean that code in library B in background process C gets the failure,…

There's also cgroup resource controls to separately govern max memory and swap usage. Thanks to systemd and systemd-run, you can easily apply and adjust them on arbitrary processes. The manpages you want are systemd.resource-control and systemd.exec. I haven't found any other equivalent tools that expose these cgroup features to the extent that systemd does.

Re: Vm.overcommit_memory=2 is the right setting for servers

#94

Post by inkyoto »

barchar wrote: post #90
inkyoto wrote: post #84

Earlier quoted context omitted.

Without going into a discussion about whether this is right or wrong, how is fork(2) wasteful? fork(2) has been copy-on-write for decades and does not copy the entire process address space. Thread stacks are a non-sequitur either as stack uses the data pages, the thread stacks are rather small in size in most scanarios, hence the thread stacks are also subject to copy-on-write. The only overhead that the use of fork(…

Unless you reserve on fork you're still over committing because after the fork writes to basically any page in either process will trigger memory commitment. Thread stacks come up because reserving them completely ahead of time would incur large amounts of memory usage. Typically they start small and grow when you touch the guards. This is a form of overcommit. Even windows dynamically grows stacks like this

> […] because after the fork writes to basically any page in either process will trigger memory commitment.

This is largely not true for most processes. For a child process to start writing into its own data pages en masse, there has to exist a specific code path that causes such behaviour. Processes do not randomly modify their own data space – it is either a bug or a peculiar workload that causes it.

You would have a stronger case if you mentioned, e.g., the JVM, which has a high complexity garbage collector (rather, multiple types of garbage collectors – each with its own behaviour), but the JVM ameliorates the problem by attempting to lock in the entire heap size at startup or bailing if it fails to do so.

In most scenarios, forking a process has a negligible effect on the overall memory consumption in the system.

Re: Vm.overcommit_memory=2 is the right setting for servers

#95

Post by kentonv »

When your system is out of memory, you do not want to return an error to the next process that allocates memory. That might be an important process, it might have nothing to do with the reason the system is out of memory, and it might not be able to gracefully handle allocation failure (realistically, most programs can't).

Instead, you want to kill the process that's hogging all the memory.

The OOM killer heuristic is not perfect, but it will generally avoid killing critical processes and is fairly good at identifying memory hogs.

And if you agree that using the OOM killer is better than returning failure to a random unlucky process, then there's no reason not to use overcommit.

Besides, overcommit is useful. Virtual-memory-based copy-on-write, allocate-on-write, sparse arrays, etc. are all useful and widely-used.

Re: Vm.overcommit_memory=2 is the right setting for servers

#96

Post by inkyoto »

barchar wrote: post #90
inkyoto wrote: post #84

Earlier quoted context omitted.

Without going into a discussion about whether this is right or wrong, how is fork(2) wasteful? fork(2) has been copy-on-write for decades and does not copy the entire process address space. Thread stacks are a non-sequitur either as stack uses the data pages, the thread stacks are rather small in size in most scanarios, hence the thread stacks are also subject to copy-on-write. The only overhead that the use of fork(…

Unless you reserve on fork you're still over committing because after the fork writes to basically any page in either process will trigger memory commitment. Thread stacks come up because reserving them completely ahead of time would incur large amounts of memory usage. Typically they start small and grow when you touch the guards. This is a form of overcommit. Even windows dynamically grows stacks like this

Also,

> Thread stacks come up because reserving them completely ahead of time would incur large amounts of memory usage. Typically they start small and grow when you touch the guards. This is a form of overcommit.

Ahead of the time memory reservation entails a page entry being allocated in the process’s page catalogue («logical» allocation), and the page «sits» dormant until it is accessed and causes a memory access fault – that is the moment when the physical allocation takes place. So copying reserved but not accessed yet pages has zero effect on the physical memory consumption of the process.

What actually happens to the thread stacks depends on the actual number of active threads. In modern designs, threads are consumed from thread pools that implement some sort of a run queue where the threads sit idle until they get assigned a unit of work. So if a thread is idle, it does not use its own stack thread and, consequently, there is no side effect on the child's COW address space.

Granted, if the child was copied with a large number of active threads, the impact will be very different.

> Even windows dynamically grows stacks like this

Windows employs a distinct process/thread design, making the UNIX concept of a process foreign. Threads are the primary construct in Windows and the kernel is highly optimised for thread management rather than processes. Cygwin has outlined significant challenges in supporting fork(2) semantics on Windows and has extensively documented the associated difficulties. However, I am veering off-topic.

Re: Vm.overcommit_memory=2 is the right setting for servers

#97

Post by Bjartr »

sidewndr46 wrote: post #58
LordGrey wrote: post #53

Earlier quoted context omitted.

malloc() and friends may always return NULL. From the man page: If successful, calloc(), malloc(), realloc(), reallocf(), valloc(), and aligned_alloc() functions return a pointer to allocated memory. If there is an error, they return a NULL pointer and set errno to ENOMEM. In practice, I find a lot of code that does not check for NULL, which is rather distressing.

It's been a while but while I agree the man page says that, my limited understanding was the typical libc on linux won't really return NULL under any sane scenario. Even when the memory can't be backed

But why would you want to violate the docs on something as fundamental as malloc? Why risk relying on implementation specific quirks in the first place?

Re: Vm.overcommit_memory=2 is the right setting for servers

#98

Post by charcircuit »

renehsz wrote: post #2

Strongly agree with this article. It highlights really well why overcommit is so harmful. Memory overcommit means that once you run out of physical memory, the OOM killer will forcefully terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software which should handle out-of-memory situations gracefully. But it feels like a lost cause…

>terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software

Having an assumption that your process will never crash is not safe. There will always be freak things like CPUs taking the wrong branch or bits randomly flipping. Parting of design a robust system is being tolerant to things like this.

Another point also mentioned is this thread is that by the time you run out of memory the system already is going to be in a bad state and now you probably don't have enough memory to even get out of it. Memory should have been freed already by telling programs to lighten up on their memory usage or by killing them and reclaiming the resources.

Re: Vm.overcommit_memory=2 is the right setting for servers

#99

Post by webstrand »

silon42 wrote: post #14
Animats wrote: post #6

Setting 2 is still pretty generous. It means "Kernel does not allow allocations that exceed swap + (RAM × overcommit_ratio / 100)." It's not a "never swap or overcommit" setting. You can still get into thrashing by memory overload. We may be entering an era when everyone in computing has to get serious about resource consumption. NVidia says GPUs are going to get more expensive for the next five years. DRAM prices ar…

For me, on the desktop, thrashing overload is the most common way the Linux system effectively crashes... (I've left it overnight a few times, sometimes it recovered, but not always). I'm not disabling overcommit for now, but maybe I should.

That thrashing is probably executable pages getting evicted, and then having to be reloaded from disk when the process resumes. Even with no swap and overcommit disabled, you'll still get thrashing before the OOM killer gets triggered.

I recommend everyone to enable linux's new multi-generational LRU, that can be configured to trigger the OOM when the workingset of the last N deciseconds doesn't fit in memory. And https://github.com/hakavlad/nohang> has some more suggestions.

Re: Vm.overcommit_memory=2 is the right setting for servers

#100

Post by themafia »

sidewndr46 wrote: post #45
LordGrey wrote: post #3

For anyone not familiar with the meaning of '2' in this context: The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.…

Do any of the settings actually result in "malloc" or a similar function returning NULL?

malloc() is an interface. There are many implementations.
Post reply on HN