Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL and the OOM killer: Why we use strict memory overcommit

ubicloud.com

Post reply on HN
Print view View original

31–40 of 140 posts

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#31

Post by geraldwhen »

furkansahin wrote: post #11
ozgune wrote: post #6

(Ozgun from Ubicloud) I agree with the blog post's technical contents, but I feel we came across too strong in the title. For Ubicloud as a managed Postgres provider, we use strict memory overcommit. Our experience with operating Postgres at scale taught us that it's better to enable this than going with the defaults. However, I can see many other scenarios, where using strict memory overcommit would have unanticipat…

(Furkan, submitter) Hmm, I haven’t thought about that. I updated the title to better reflect Ubicloud Postgres' position.

Is this an AI response?

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#32

Post by toast0 »

szmarczak wrote: post #28
0x1d7 wrote: post #23

Earlier quoted context omitted.

This is almost always a bad idea. If no memory is available where a page file would make a difference, this leads to application crashes instead. A crash is (usually) worse than paging. Certain applications, Photoshop being the historical example, will outright fail to run with no page file present.

> this leads to application crashes instead Same happens if the page file is full. In that case, why don't those programs use disk directly instead? No such problem would've ever occured if programs hadn't allocated more than they actually use.

By default, windows uses an expandable page file.

Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this.

> No such problem would've ever occured if programs hadn't allocated more than they actually use.

That's part of the issue, but sometimes things do in fact use too much memory as well as allocate too much.

Another part of the issue is that few programs are built to handle allocation failures.

And then you have a metrics issue. There's not really a good metric to know when you're out of memory, other than performance collapse. If your applications don't use disk, it's not too hard; but when they do use disk, performance will collapse once there's insufficient memory to provide the disk caching needed. In my experience, adding a small swap and monitoring swap i/o can be pretty helpful, and a small swap doesn't tend to allow long thrashing when memory use grows. But that's not universal and everybody loves to hate swap these days.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#33

Post by furkansahin »

geraldwhen wrote: post #31
furkansahin wrote: post #11

Earlier quoted context omitted.

(Furkan, submitter) Hmm, I haven’t thought about that. I updated the title to better reflect Ubicloud Postgres' position.

Is this an AI response?

No.. But I have been using a lot of AI, recently. It might have impacted how I form my phrases? maybe?

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#34

Post by szmarczak »

0x1d7 wrote: post #30
szmarczak wrote: post #28

Earlier quoted context omitted.

> this leads to application crashes instead Same happens if the page file is full. In that case, why don't those programs use disk directly instead? No such problem would've ever occured if programs hadn't allocated more than they actually use.

Your argument falls flat when a page file can be multi-GB and automatically grow. And if your application admin was competent, memory monitoring would be part of the application monitoring stack. An application that grows in such a way (besides having backing stores for memory-mapped files, as well) will often perform so poorly that it requires addressing (adding RAM, looking for application faults, etc). A page file…

> memory monitoring would be part of the application monitoring stack

You don't need it if you have everything allocated upfront. TigerBeetle does this, everybody else can.

Using something like Rust is already a huge win when compared to shipping a browser or running Node.js.

> Your argument falls flat when a page file can be multi-GB and automatically grow

This doesn't solve the original issue and only masks the underlying problem.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#35

Post by leononame »

xyzzy_plugh wrote: post #14
leononame wrote: post #4

This has bitten me multiple times. The problem I have is that at work we deploy the application (written in Go) and PostgreSQL on the same machine. The backend app allocates a lot of virtual memory, and initially we had overcommit to 0 (heuristic). This caused crashes on big queries in PostgreSQL and we set it to 2. The whole system became a bit unstable because the backend would still allocate a lot of virtual memor…

I'm not sure if you are aware but there are relatively recent environment variables you can set to help contain Go memory to a fixed size. GOMEMLIMIT works very well if you set it to around 90% of available memory as a rough heuristic. You should definitely profile your application to fine tune this number (e.g. if you link with C libraries that hold large memory pools then Go doesn't account for that) but also to id…

I was aware of GOMEMLIMIT, but it didn't cross my mind in this case, thanks for pointing it out. It could be really useful! I'll have to check our specific use case

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#36

Post by man8alexd »

tredre3 wrote: post #27
swinglock wrote: post #22

Earlier quoted context omitted.

I don't think it has overcommit at all, at least that's the default. That would be why you don't have Windows OOM killer stories.

The reason you hear less about Window's OOM killer is simply because it works well. The Linux Kernel OOM killer kills random things. Userspace OOM killers are meant to improve this, and they work well in a server situation when you already know in advance what is likely to go haywire and what is safe to kill. But they don't work well on desktop (some of them are improving but it doesn't seem to be a priority). The Wi…

> The Linux Kernel OOM killer kills random things.

By default, the Linux kernel kills the largest process in the system (unless OOM adjust was applied).

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#37

Post by quotemstr »

toast0 wrote: post #32
szmarczak wrote: post #28

Earlier quoted context omitted.

> this leads to application crashes instead Same happens if the page file is full. In that case, why don't those programs use disk directly instead? No such problem would've ever occured if programs hadn't allocated more than they actually use.

By default, windows uses an expandable page file. Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this. > No such problem would've ever occured if programs hadn't allocated more than they actually use. That's part of the issue, but somet…

> Typically, performance drops enough that the user kills the program or reboots before the page file expands to fill the disk. And other threads here suggest there is something that will prompt users to kill programs in states like this.

Not in the age of NVMe it doesn't. Swap is fast now. Plus, at least on Linux, you can put zswap in front of the regular swap and introduce an even faster level of memory hierarchy and thereby make page-outs even more profitable.

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#38

Post by man8alexd »

szmarczak wrote: post #3

I have disabled overcommit both on Windows and on Linux. I hate having random programs being killed. Unfortunately, many programs commit 2x memory than they actually use. Often I see ~32GB committed and ~16GB resident.

disabling overcommit is trading OOM for random program crashes due to the inability to handle ENOMEM. It also wastes a lot of system memory.

https://unix.stackexchange.com/questions/797835/disabling-ov...

Re: PostgreSQL and the OOM killer: Why we use strict memory overcommit

#40

Post by fpoling »

leononame wrote: post #4

This has bitten me multiple times. The problem I have is that at work we deploy the application (written in Go) and PostgreSQL on the same machine. The backend app allocates a lot of virtual memory, and initially we had overcommit to 0 (heuristic). This caused crashes on big queries in PostgreSQL and we set it to 2. The whole system became a bit unstable because the backend would still allocate a lot of virtual memor…

You can also use cgroup to set allocation policy for a particular program.
Post reply on HN