Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL and the OOM killer: Why we use strict memory overcommit

ubicloud.com

Post reply on HN
Print view View original

61–70 of 140 posts

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#61

Post by senfiaj »

wongarsu wrote: post #20
nok22kon wrote: post #16

Earlier quoted context omitted.

how exactly did you disabled it on Windows? I dont think it has an option for that.

Not overcommitting is Windows's default and only behavior A memory allocator can implement overcommit, because you can separate reserving virtual memory and having it backed by physical memory into two different system calls. But from the point of view of the kernel, any time it promises to give you physical memory that memory is backed either by RAM or by space reserved in the swap file

As I understand, Windows can also lazily allocate pages, but it does after making sure the memory budget is adequate in a case of low physical RAM pressure and is guaranteed to be backed up by a page. But yeah, Linux approach is really sloppy.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#62

Post by senfiaj »

IshKebab wrote: post #57
grg0 wrote: post #52

Earlier quoted context omitted.

What exactly does Rust solve here? Virtual memory is a hardware/OS feature.

The bug that is detailed in the article. Wouldn't have happened with Rust.

There was also a bug in the Linux kernel, or did I miss something?

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#63

Post by fdr »

man8alexd wrote: post #46

Mode 0 (Heuristic) is described incorrectly. All this complex heuristic was removed almost a decade ago. Currently, the kernel refuses a single allocation that exceeds the physical memory. That is all. The article ignores the proper modern solution to prevent OOM killing of critical processes - OOM Score Adjust. Tuning CommitLimit manually is an archaic, imprecise, and error-prone way to handle memory limits, only su…

The key thing is Postgres does handle enomem well and does a nice rollback rather than crashing the server and entering crash recovery. It’s one of the few programs that does. Exceptions for the exception.

Even a revised heuristic that only spots large, individual allocations is not going to do the job.

Oom score adjust also doesn’t do the job: because the only interesting workload is Postgres, if a backend does a page fault that needs memory, who dies? Another sibling Postgres, almost certainly. Then postmaster does crash recovery, which most would rather avoid. High performance databases with distant checkpoints can take a while to come back up.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#64

Post by swinglock »

quotemstr wrote: post #37
toast0 wrote: post #32

Earlier quoted context omitted.

By default, windows uses an expandable page file. Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this. > No such problem would've ever occured if programs hadn't allocated more than they actually use. That's part of the issue, but somet…

> Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this. Not in the age of NVMe it doesn't. Swap is fast now. Plus, at least on Linux, you can put zswap in front of the regular swap and introduce an even faster level of memory hierarchy a…

Windows does memory compression too.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#65

Post by grg0 »

IshKebab wrote: post #57
grg0 wrote: post #52

Earlier quoted context omitted.

What exactly does Rust solve here? Virtual memory is a hardware/OS feature.

The bug that is detailed in the article. Wouldn't have happened with Rust.

This is not a memory safety bug, but a bug resulting from a type coercion of int to bool. I don't know if Rust is stricter but your original statement was confusing.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#66

Post by man8alexd »

fdr wrote: post #63
man8alexd wrote: post #46

Mode 0 (Heuristic) is described incorrectly. All this complex heuristic was removed almost a decade ago. Currently, the kernel refuses a single allocation that exceeds the physical memory. That is all. The article ignores the proper modern solution to prevent OOM killing of critical processes - OOM Score Adjust. Tuning CommitLimit manually is an archaic, imprecise, and error-prone way to handle memory limits, only su…

The key thing is Postgres does handle enomem well and does a nice rollback rather than crashing the server and entering crash recovery. It’s one of the few programs that does. Exceptions for the exception. Even a revised heuristic that only spots large, individual allocations is not going to do the job. Oom score adjust also doesn’t do the job: because the only interesting workload is Postgres, if a backend does a pa…

They do have sidecars like prometheus, node_exporter running alongside Postgres and they include them in their MemoryLimit calculations.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#67

Post by IshKebab »

senfiaj wrote: post #62
IshKebab wrote: post #57

Earlier quoted context omitted.

The bug that is detailed in the article. Wouldn't have happened with Rust.

There was also a bug in the Linux kernel, or did I miss something?

Yes there was. It's detailed in the article.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#68

Post by senderista »

The proper way to handle OOM is to do what mature databases do: implement your own memory accounting, use only your own allocators integrated with the accounting system, and ensure that every allocation path can recover from OOM. Easier said than done.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#69

Post by man8alexd »

IshKebab wrote: post #58
man8alexd wrote: post #50

Earlier quoted context omitted.

Yes, many have tried to use strict overcommit on the desktop. It is a good footgun. https://unix.stackexchange.com/a/797888/1027

That doesn't really explain why it is a footgun.

The last paragraph:

> On the modern desktop, where programmers don't care about failing malloc(), disabling overcommit is shooting yourself in the foot. As you can observe, the memory allocations start failing long before the memory is exhausted.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#70

Post by man8alexd »

senderista wrote: post #68

The proper way to handle OOM is to do what mature databases do: implement your own memory accounting, use only your own allocators integrated with the accounting system, and ensure that every allocation path can recover from OOM. Easier said than done.

MariaDB recently implemented memory PSI monitoring but failed with that in a curious way and disabled it afterwards by default. The failure is that under memory pressure, they flushed the entire InnoDB buffer pool.
Post reply on HN