Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL and the OOM killer: Why we use strict memory overcommit

ubicloud.com

Post reply on HN
Print view View original

101–110 of 140 posts

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#101

Post by leononame »

man8alexd wrote: post #51
leononame wrote: post #49

Earlier quoted context omitted.

Interesting idea. How does that work in practice? If I've got 64GiB RAM, and PostgreSQL has 32GiB memory usage, and Go has 32GiB of memory. If the database requests more memory, it gets ENOMEM, but if the backend app requests more memory, it does get some more because it can overcommit? Sounds dangerous, if the go program then writes to the overcommitted memory, you'd still trigger the OOM killer, right?

cgroups have nothing to do with overcommit and memory allocation. They limit actual memory usage for a specific program or group of programs. If this program tries to use more memory than the cgroup memory limit, the program gets OOM killed.

Ok, got it, thanks. The wording in the comment I was responding to confused me and made me think it was possible to also change the overcommit behavior for a cgroup, not just the limits.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#102

Post by IshKebab »

grg0 wrote: post #65
IshKebab wrote: post #57

Earlier quoted context omitted.

The bug that is detailed in the article. Wouldn't have happened with Rust.

This is not a memory safety bug, but a bug resulting from a type coercion of int to bool. I don't know if Rust is stricter but your original statement was confusing.

Exactly my point. Lots of people think Rust only prevents memory safety bugs, but this is an example of a bug that isn't a memory safety bug but also wouldn't have happened in Rust.

They used an int with special meanings for negative/0/positive values. Very common in C, and not at all type safe (all meanings have the same type). In Rust you would use an enum or Result, it would be type safe and the refactoring mistake they made would have been a compile time error.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#103

Post by IshKebab »

senfiaj wrote: post #71
IshKebab wrote: post #67

Earlier quoted context omitted.

Yes there was. It's detailed in the article.

But why wouldn't have happened with Rust? Sorry I can't find anything about Rust in the article. Or you mean if the Linux kernel was written in Rust and that stupid bool coercion was not possible?

Because Rust programmers would have used an enum or Result.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#104

Post by IshKebab »

man8alexd wrote: post #69
IshKebab wrote: post #58

Earlier quoted context omitted.

That doesn't really explain why it is a footgun.

The last paragraph: > On the modern desktop, where programmers don't care about failing malloc(), disabling overcommit is shooting yourself in the foot. As you can observe, the memory allocations start failing long before the memory is exhausted.

If a memory allocations fails with strict mode then you'll get a null pointer returns and some kind of crash or panic (in code that doesn't handle it properly).

If it fails with the default mode the whole process will get killed by the OS. Is that really much better?

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#105

Post by man8alexd »

zbentley wrote: post #97
fpoling wrote: post #44

Earlier quoted context omitted.

Java allowed to handle out-of-memory rather well if one wanted to even 25 years ago. Basically one allocated a buffer on a startup taking 5% of memory that the application was supposed to use and made all threads to catch the oom exception. When handling that the buffer would be released, GC would be forced and a special flag would be set asking app to cancel any memory-intensive tasks until enough memory would be re…

Yeah, crash ballast is an extremely underrated tool. It also works for the OOM killer: run a daemon with a child process that holds some fixed amount of memory. Adjust OOM scores of everything else on the system lower than the child. If the parent’s waitpid() returns due to an OOM kill, send an alert/shutdown nonessentials/sync buffers to disk and so on.

Maybe you should skip running a useless child process and just use PSI to monitor memory pressure.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#106

Post by man8alexd »

frollogaston wrote: post #100
mono442 wrote: post #55

The problem with disabling the memory overcommit is that then the RAM is wasted. That can be worked around with setting up swap but then the disk space is wasted.

Why are programs even allocating memory that they don't use?

Because no one can predict the future, and they don't know how many resources they will need.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#107

Post by man8alexd »

zbentley wrote: post #95
man8alexd wrote: post #70

Earlier quoted context omitted.

MariaDB recently implemented memory PSI monitoring but failed with that in a curious way and disabled it afterwards by default. The failure is that under memory pressure, they flushed the entire InnoDB buffer pool.

The issue is that there’s no generally correct behavior. Should a database under memory pressure stay up at all costs even if it becomes unusably slow (by e.g. nuking 99% of the buffer cache)? Or should it crash/failover hard with a likelihood of potential recovery afterwards, even if it technically could have stayed up? Something in between? There’s no correct-in-general answers to those questions. This is a hard pr…

In this specific case, the correct behaviour would be to drop a part of the buffer pool until the memory pressure is gone. The context-dependent question is how much and how fast to drop. The current implementation drops to a single configurable level but I suspect it could have implemented better heuristics.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#108

Post by man8alexd »

koverstreet wrote: post #48
baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

The mm people are increasingly hostile to any method of handling OOMs (like, just failing the allocation) besides the OOM killer - it's become very dominated by the hyperscalars and cloud vendors. Working around mm nuttiness is a frequent source of frustration.

There is no point in managing memory allocations as they have little relation to actual memory usage. There are also other methods than the OOM killer to handle OOM, like process throttling using cgroups "memory.high" limits.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#109

Post by man8alexd »

IshKebab wrote: post #104
man8alexd wrote: post #69

Earlier quoted context omitted.

The last paragraph: > On the modern desktop, where programmers don't care about failing malloc(), disabling overcommit is shooting yourself in the foot. As you can observe, the memory allocations start failing long before the memory is exhausted.

If a memory allocations fails with strict mode then you'll get a null pointer returns and some kind of crash or panic (in code that doesn't handle it properly). If it fails with the default mode the whole process will get killed by the OS. Is that really much better?

OOM is better. If a program doesn't handle ENOMEM properly, then its state is unpredictable and can lead to data corruption.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#110

Post by psd1 »

cmurf wrote: post #93
koverstreet wrote: post #48

Earlier quoted context omitted.

The mm people are increasingly hostile to any method of handling OOMs (like, just failing the allocation) besides the OOM killer - it's become very dominated by the hyperscalars and cloud vendors. Working around mm nuttiness is a frequent source of frustration.

Kernel oomkiller is useless for the desktop. It only cares about kernel survival. There's user space oom options, like oomd, earlyoom ... but I'm not sure any heuristic really knows what user space program to clobber. Resource control via cgroupsv2 sounds better for both desktop and server use cases, differing by what processes receive minimum resources. But IO isolation remains elusive. Some suggest a database of ha…

> Well before it even considers the situation human interactivity with the system was lost.

"Are you logged on to DB1?"

"...yes?"

"What did you do, it just died"

This has happened multiple times

Post reply on HN