Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL and the OOM killer: Why we use strict memory overcommit

ubicloud.com

Post reply on HN
Print view View original

81–90 of 140 posts

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#81

Post by layer8 »

koverstreet wrote: post #48
baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

The mm people are increasingly hostile to any method of handling OOMs (like, just failing the allocation) besides the OOM killer - it's become very dominated by the hyperscalars and cloud vendors. Working around mm nuttiness is a frequent source of frustration.

mm?

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#82

Post by 0x1d7 »

quotemstr wrote: post #37
toast0 wrote: post #32

Earlier quoted context omitted.

By default, windows uses an expandable page file. Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this. > No such problem would've ever occured if programs hadn't allocated more than they actually use. That's part of the issue, but somet…

> Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this. Not in the age of NVMe it doesn't. Swap is fast now. Plus, at least on Linux, you can put zswap in front of the regular swap and introduce an even faster level of memory hierarchy a…

Swap is not fast. Faster than it was, yes, fast, no.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#83

Post by 0x1d7 »

szmarczak wrote: post #34
0x1d7 wrote: post #30

Earlier quoted context omitted.

Your argument falls flat when a page file can be multi-GB and automatically grow. And if your application admin was competent, memory monitoring would be part of the application monitoring stack. An application that grows in such a way (besides having backing stores for memory-mapped files, as well) will often perform so poorly that it requires addressing (adding RAM, looking for application faults, etc). A page file…

> memory monitoring would be part of the application monitoring stack You don't need it if you have everything allocated upfront. TigerBeetle does this, everybody else can. Using something like Rust is already a huge win when compared to shipping a browser or running Node.js. > Your argument falls flat when a page file can be multi-GB and automatically grow This doesn't solve the original issue and only masks the und…

> This doesn't solve the original issue and only masks the underlying problem.

You're moving goal posts. No, a page file doesn't solve the problem of a misbehaving application, but it does solve the problem of an app crash because no more VAS allocation can be made.

You should really dive into Windows Internals. Only misinformed gamers turn off page files.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#84

Post by 0x1d7 »

man8alexd wrote: post #36
tredre3 wrote: post #27

Earlier quoted context omitted.

The reason you hear less about Window's OOM killer is simply because it works well. The Linux Kernel OOM killer kills random things. Userspace OOM killers are meant to improve this, and they work well in a server situation when you already know in advance what is likely to go haywire and what is safe to kill. But they don't work well on desktop (some of them are improving but it doesn't seem to be a priority). The Wi…

> The Linux Kernel OOM killer kills random things. By default, the Linux kernel kills the largest process in the system (unless OOM adjust was applied).

Which, by default, is dumb for a presumably interactive system. Photoshop (or equiv) is going to be my "largest" process on a system. Because it's the one I'm interacting with.

Don't kill what I'm using.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#85

Post by 0x1d7 »

senfiaj wrote: post #53
swinglock wrote: post #22

Earlier quoted context omitted.

I don't think it has overcommit at all, at least that's the default. That would be why you don't have Windows OOM killer stories.

I think Windows also uses something similar to Linux memory overcommit (maybe call it "lazy page allocation"), but probably all the available virtual memory is backed by the page file, the OS has limit for the allocation size and all the edge cases are handled well under low memory conditions. For example, think about stack memory. Megabytes of stack space can be reserved for each thread, but since programs rarely us…

Windows will never overcommit. All pages are backed by physical memory or backing store.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#86

Post by layer8 »

man8alexd wrote: post #39
wongarsu wrote: post #21

For once, Microsoft's decision to just not do overcommit in Windows seems sensible

Windows doesn't need to fork, and you can't fork a large process without overcommit.

One could rephrase the parent that Windows did the sensible thing by not using the fork model.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#87

Post by callahad »

layer8 wrote: post #81
koverstreet wrote: post #48

Earlier quoted context omitted.

The mm people are increasingly hostile to any method of handling OOMs (like, just failing the allocation) besides the OOM killer - it's become very dominated by the hyperscalars and cloud vendors. Working around mm nuttiness is a frequent source of frustration.

mm?

I believe koverstreet is referring to the Linux kernel Memory Management subsystem (https://docs.kernel.org/mm/index.html)

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#88

Post by man8alexd »

0x1d7 wrote: post #84
man8alexd wrote: post #36

Earlier quoted context omitted.

> The Linux Kernel OOM killer kills random things. By default, the Linux kernel kills the largest process in the system (unless OOM adjust was applied).

Which, by default, is dumb for a presumably interactive system. Photoshop (or equiv) is going to be my "largest" process on a system. Because it's the one I'm interacting with. Don't kill what I'm using.

You can always tell the system not to kill your Photoshop (or whatever else) by setting the OOM Score Adjust. This mechanism has existed for almost two decades and systemd has supported it for over a decade.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#89

Post by senfiaj »

0x1d7 wrote: post #85
senfiaj wrote: post #53

Earlier quoted context omitted.

I think Windows also uses something similar to Linux memory overcommit (maybe call it "lazy page allocation"), but probably all the available virtual memory is backed by the page file, the OS has limit for the allocation size and all the edge cases are handled well under low memory conditions. For example, think about stack memory. Megabytes of stack space can be reserved for each thread, but since programs rarely us…

Windows will never overcommit. All pages are backed by physical memory or backing store.

Yes, that was my point. But Windows might still allocate pages lazily (i.e. reserve them and actually allocate only when the program writes into them). The difference from Linux is that it will only reserve if the allocation has a reasonable size and the total committed allocation doesn't exceed certain safe limits / quotas, so it will be possible to safely swap on disk later under low memory conditions.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#90

Post by pyinstallwoes »

baq wrote: post #45

Linux vm defaults are legit insane in 2026. - system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now) - system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose) - system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer - memory compression…

What are better defaults for 2026+?
Post reply on HN