Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL and the OOM killer: Why we use strict memory overcommit

ubicloud.com

Post reply on HN
Print view View original

51–60 of 140 posts

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#51

Post by man8alexd »

leononame wrote: post #49
fpoling wrote: post #40

Earlier quoted context omitted.

You can also use cgroup to set allocation policy for a particular program.

Interesting idea. How does that work in practice? If I've got 64GiB RAM, and PostgreSQL has 32GiB memory usage, and Go has 32GiB of memory. If the database requests more memory, it gets ENOMEM, but if the backend app requests more memory, it does get some more because it can overcommit? Sounds dangerous, if the go program then writes to the overcommitted memory, you'd still trigger the OOM killer, right?

cgroups have nothing to do with overcommit and memory allocation. They limit actual memory usage for a specific program or group of programs. If this program tries to use more memory than the cgroup memory limit, the program gets OOM killed.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#52

Post by grg0 »

IshKebab wrote: post #47

There's so much great stuff here. First, Linux's default memory management strategy is bonkers. OOM killing rarely actually works in my experience, at least on desktop. It takes ages to kick in and usually the system just freezes and you have to hard reboot. I've experienced this on every Linux system I've used, even my current one with 128GB of RAM and 64GB of swap, so don't say "it works for me". Windows and Mac do…

What exactly does Rust solve here? Virtual memory is a hardware/OS feature.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#53

Post by senfiaj »

swinglock wrote: post #22
nok22kon wrote: post #16

Earlier quoted context omitted.

how exactly did you disabled it on Windows? I dont think it has an option for that.

I don't think it has overcommit at all, at least that's the default. That would be why you don't have Windows OOM killer stories.

I think Windows also uses something similar to Linux memory overcommit (maybe call it "lazy page allocation"), but probably all the available virtual memory is backed by the page file, the OS has limit for the allocation size and all the edge cases are handled well under low memory conditions. For example, think about stack memory. Megabytes of stack space can be reserved for each thread, but since programs rarely use the whole stack, it's not rational to immediately allocate all the physical pages, it will waste a lot of memory (especially for multi-threaded programs). The stack grows on demand when the program hits a guard page.

In short, Windows partially does the same lazy thing but unlike Linux with its optimistic overcommit, it is stricter about commit/backing-budget reservation.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#54

Post by 10000truths »

I'd be interested to see a Linux distribution whose entire shtick is to run well-behaved under a kernel with overcommit disabled. But it would be a huge undertaking. Besides the obvious issue with fork(), there are a lot of programs and libraries out there that implicitly rely on overcommit due to not checking malloc() for failure.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#56

Post by senfiaj »

tredre3 wrote: post #27
swinglock wrote: post #22

Earlier quoted context omitted.

I don't think it has overcommit at all, at least that's the default. That would be why you don't have Windows OOM killer stories.

The reason you hear less about Window's OOM killer is simply because it works well. The Linux Kernel OOM killer kills random things. Userspace OOM killers are meant to improve this, and they work well in a server situation when you already know in advance what is likely to go haywire and what is safe to kill. But they don't work well on desktop (some of them are improving but it doesn't seem to be a priority). The Wi…

I think Windows doesn't do OOM kill, it just fails fast for unreasonably large memory allocations. If the allocation is succeeds, the committed virtual pages are backed up (or more precisely guaranteed to be backed up) by a page file.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#57

Post by IshKebab »

grg0 wrote: post #52
IshKebab wrote: post #47

There's so much great stuff here. First, Linux's default memory management strategy is bonkers. OOM killing rarely actually works in my experience, at least on desktop. It takes ages to kick in and usually the system just freezes and you have to hard reboot. I've experienced this on every Linux system I've used, even my current one with 128GB of RAM and 64GB of swap, so don't say "it works for me". Windows and Mac do…

What exactly does Rust solve here? Virtual memory is a hardware/OS feature.

The bug that is detailed in the article. Wouldn't have happened with Rust.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#58

Post by IshKebab »

man8alexd wrote: post #50
IshKebab wrote: post #47

There's so much great stuff here. First, Linux's default memory management strategy is bonkers. OOM killing rarely actually works in my experience, at least on desktop. It takes ages to kick in and usually the system just freezes and you have to hard reboot. I've experienced this on every Linux system I've used, even my current one with 128GB of RAM and 64GB of swap, so don't say "it works for me". Windows and Mac do…

Yes, many have tried to use strict overcommit on the desktop. It is a good footgun. https://unix.stackexchange.com/a/797888/1027

That doesn't really explain why it is a footgun.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#59

Post by man8alexd »

10000truths wrote: post #54

I'd be interested to see a Linux distribution whose entire shtick is to run well-behaved under a kernel with overcommit disabled. But it would be a huge undertaking. Besides the obvious issue with fork(), there are a lot of programs and libraries out there that implicitly rely on overcommit due to not checking malloc() for failure.

There is some kind of illusion or myth that strict overcommit solves memory management issues.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#60

Post by sisve »

furkansahin wrote: post #33
geraldwhen wrote: post #31

Earlier quoted context omitted.

Is this an AI response?

No.. But I have been using a lot of AI, recently. It might have impacted how I form my phrases? maybe?

Nah, your doing great, you just reflect on you own position and adjust it. People are just (too) suspicious when people are not locked in their ways and are nice instead of hostile
Post reply on HN