Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL and the OOM killer: Why we use strict memory overcommit

ubicloud.com

Post reply on HN
Print view View original

91–100 of 140 posts

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#91

Post by [deleted] »

koverstreet wrote: post #48
baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

The mm people are increasingly hostile to any method of handling OOMs (like, just failing the allocation) besides the OOM killer - it's become very dominated by the hyperscalars and cloud vendors. Working around mm nuttiness is a frequent source of frustration.

[deleted]

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#92

Post by alexrp »

baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

> Both Windows and macOS do so much better out of the box for essentially any workload.

We test FreeBSD, Linux, macOS, NetBSD, OpenBSD, and Windows in Zig's CI fleet. Of these, Windows is the only OS that we've had to configure with swap double the size of physical RAM to not hit completely unjustifiable OOMs.

By "unjustifiable", I mean that we're not even close to actually running out of physical memory (let alone swap), but the MM seems to be doing a horrible job of making unused memory actually available to processes.

It's possible there's a relevant configuration knob here that we're just not aware of... but the point is, the default behavior does in fact suck.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#93

Post by cmurf »

koverstreet wrote: post #48
baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

The mm people are increasingly hostile to any method of handling OOMs (like, just failing the allocation) besides the OOM killer - it's become very dominated by the hyperscalars and cloud vendors. Working around mm nuttiness is a frequent source of frustration.

Kernel oomkiller is useless for the desktop. It only cares about kernel survival. There's user space oom options, like oomd, earlyoom ... but I'm not sure any heuristic really knows what user space program to clobber.

Resource control via cgroupsv2 sounds better for both desktop and server use cases, differing by what processes receive minimum resources. But IO isolation remains elusive. Some suggest a database of hardware capabilities, I wonder if a cheap estimator of throughput and latencies could do this dynamically.

Anyway, I'm not convinced user space should care about or rely on the kernel oomkiller. Well before it even considers the situation human interactivity with the system was lost.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#94

Post by LoganDark »

alexrp wrote: post #92
baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

> Both Windows and macOS do so much better out of the box for essentially any workload. We test FreeBSD, Linux, macOS, NetBSD, OpenBSD, and Windows in Zig's CI fleet. Of these, Windows is the only OS that we've had to configure with swap double the size of physical RAM to not hit completely unjustifiable OOMs. By "unjustifiable", I mean that we're not even close to actually running out of physical memory (let alone s…

Windows seems to work a lot better with a 16MB page file for whatever reason, just because it refuses to enable memory compression without it. Fucking stupid

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#95

Post by zbentley »

man8alexd wrote: post #70
senderista wrote: post #68

The proper way to handle OOM is to do what mature databases do: implement your own memory accounting, use only your own allocators integrated with the accounting system, and ensure that every allocation path can recover from OOM. Easier said than done.

MariaDB recently implemented memory PSI monitoring but failed with that in a curious way and disabled it afterwards by default. The failure is that under memory pressure, they flushed the entire InnoDB buffer pool.

The issue is that there’s no generally correct behavior. Should a database under memory pressure stay up at all costs even if it becomes unusably slow (by e.g. nuking 99% of the buffer cache)? Or should it crash/failover hard with a likelihood of potential recovery afterwards, even if it technically could have stayed up? Something in between?

There’s no correct-in-general answers to those questions. This is a hard problem due to context dependence; that’s why there are so many knobs.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#96

Post by zbentley »

10000truths wrote: post #54

I'd be interested to see a Linux distribution whose entire shtick is to run well-behaved under a kernel with overcommit disabled. But it would be a huge undertaking. Besides the obvious issue with fork(), there are a lot of programs and libraries out there that implicitly rely on overcommit due to not checking malloc() for failure.

Sigh. Malloc failure should have had to be trapped with a signal or something, not just a return status. I know, I know, threads and nesting handlers make that hard and historical precedent makes it impossible to retrofit, but I can still dream.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#97

Post by zbentley »

fpoling wrote: post #44
sterwill wrote: post #10

Earlier quoted context omitted.

Does this result in programs more frequently erroring/crashing because they can't allocate? I don't know how well many of the programs I frequently use on my desktop (Firefox, GNOME desktop, JVM + IntelliJ, Slack, etc.) handle allocation failures. I'm not sure they would do much better than crash, but I know the default OOM killer settings work well for me. About once a year a real runaway process (usually a throwawa…

Java allowed to handle out-of-memory rather well if one wanted to even 25 years ago. Basically one allocated a buffer on a startup taking 5% of memory that the application was supposed to use and made all threads to catch the oom exception. When handling that the buffer would be released, GC would be forced and a special flag would be set asking app to cancel any memory-intensive tasks until enough memory would be re…

Yeah, crash ballast is an extremely underrated tool.

It also works for the OOM killer: run a daemon with a child process that holds some fixed amount of memory. Adjust OOM scores of everything else on the system lower than the child. If the parent’s waitpid() returns due to an OOM kill, send an alert/shutdown nonessentials/sync buffers to disk and so on.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#98

Post by [deleted] »

nok22kon wrote: post #29
tredre3 wrote: post #27

Earlier quoted context omitted.

The reason you hear less about Window's OOM killer is simply because it works well. The Linux Kernel OOM killer kills random things. Userspace OOM killers are meant to improve this, and they work well in a server situation when you already know in advance what is likely to go haywire and what is safe to kill. But they don't work well on desktop (some of them are improving but it doesn't seem to be a priority). The Wi…

damn, good observation, when my data analysis python script goes wrong and allocates 24 GB of RAM on a 32 GB computer, it crashes (gets killed) with "out of memory" error. I've never seen something else getting killed

[deleted]

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#99

Post by frollogaston »

nok22kon wrote: post #29
tredre3 wrote: post #27

Earlier quoted context omitted.

The reason you hear less about Window's OOM killer is simply because it works well. The Linux Kernel OOM killer kills random things. Userspace OOM killers are meant to improve this, and they work well in a server situation when you already know in advance what is likely to go haywire and what is safe to kill. But they don't work well on desktop (some of them are improving but it doesn't seem to be a priority). The Wi…

damn, good observation, when my data analysis python script goes wrong and allocates 24 GB of RAM on a 32 GB computer, it crashes (gets killed) with "out of memory" error. I've never seen something else getting killed

That's what I've seen Linux do too. What's different about how Windows chooses a process to kill?

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#100

Post by frollogaston »

mono442 wrote: post #55

The problem with disabling the memory overcommit is that then the RAM is wasted. That can be worked around with setting up swap but then the disk space is wasted.

Why are programs even allocating memory that they don't use?
Post reply on HN