PostgreSQL, Memory and the Cloud
sosna.de
PostgreSQL, Memory and the Cloud
1–10 of 60 posts
Re: PostgreSQL, Memory and the Cloud
#2So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
Re: PostgreSQL, Memory and the Cloud
#3So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
And if there are no problems, why is it not disabled by default?
Re: PostgreSQL, Memory and the Cloud
#4So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
And if there are no problems, why is it not disabled by default?
If you mean “why isn’t it disabled by default on Linux installs”: most programs don’t expect malloc(2) to ever return NULL. Those programs will just assume the return value from malloc(2) is valid memory. In the best case, they’ll immediately write to it and protection-fault. In the worst case, they’ll hold onto this NULL pointer for a while, passing it around, until eventually something else somewhere distant in the program blows up some unknown amount of time later.
Even those programs that are “malloc(2) error aware”, often do something stupid and counterproductive in response, like attempting to allocate more memory for an exception object / stack trace / error string.
Programs that do something useful in response to a NULL malloc(2) return result — useful for the stability of the system as a whole, better than what the OOM killer gets you — are rare, even on servers. Usually it’s only stateful, long-running, DBMS-like daemons that 1. bother, and 2. have the engineering effort put into them to do the right thing.
Re: PostgreSQL, Memory and the Cloud
#5So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
Some programs allocate a lot of virtual memory and then don't use it.
Also, linux's forking model can result in a lot of virtual memory being allocated if a heavy-weight program tries to fork+exec a lot of smaller programs, since fork+exec it not atomic and briefly doubles the virtual memory usage of the original program.
I think there are better ways to spawn programs that don't suffer from this problem now...
If you have programs that are written to allocate virtual memory sparingly (like postgres) then that should be fine.
However, there is a second way you can be caught out: even if you disable overcommit, your program can still be OOM killed for violating cgroup limits, since cgroup limits always behave as though over-commit is enabled (ie. they allow you to allocate more than you are allowed, and then you get OOM killed when you try to use the allocated memory). This means you'd have to be really careful running eg. postgres inside a kubernetes pod.
This behaviour really sucks IMO. I would like it if you could set overcommit on a per-program basis, so that eg. postgres can say "I know what I'm doing - when I allocate virtual memory I want you to really allocate it (and tell me now if you can't...)". I think you can somewhat achieve this with memory locking, but that prevents it from being paged out at all...
Re: PostgreSQL, Memory and the Cloud
#6Earlier quoted context omitted.
And if there are no problems, why is it not disabled by default?
If you mean “why isn’t it disabled by default on Linux installs”: most programs don’t expect malloc(2) to ever return NULL. Those programs will just assume the return value from malloc(2) is valid memory. In the best case, they’ll immediately write to it and protection-fault. In the worst case, they’ll hold onto this NULL pointer for a while, passing it around, until eventually something else somewhere distant in the…
I think your first paragraph is too pessimistic.
You can get a decent idea of the behaviour of the programs you use when they run out of memory by running under 'ulimit -v' with a low limit.
In my experience most (though far from all) of the programs I use managed a clean abort with an error message (as from a traditional xmalloc()).
Re: PostgreSQL, Memory and the Cloud
#7So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
And if there are no problems, why is it not disabled by default?
Because many program "unnecessarily" allocate a lot of virtual memory and then never actually page it in, disabling overcommit will start killing processes due to lack of memory even though most of the memory isn't been actually used.
Re: PostgreSQL, Memory and the Cloud
#8Wow, the title of this post is very calm compared to what is actually happening.
CloudSQL Postgres is running with a misconfigured OS OOM killer, crashes Postmaster randomly even if memory use is below instance spec. GCP closes this bug report as "Won't fix".
This is a priority 1 issue. Seeing a wontfix for this has completely destroyed my trust of their judgement. The bug report states that they have been in contact with support since February.
Unbelievable attitude towards fixing production critical problems of their platform affecting all customers.
Re: PostgreSQL, Memory and the Cloud
#9Earlier quoted context omitted.
If you mean “why isn’t it disabled by default on Linux installs”: most programs don’t expect malloc(2) to ever return NULL. Those programs will just assume the return value from malloc(2) is valid memory. In the best case, they’ll immediately write to it and protection-fault. In the worst case, they’ll hold onto this NULL pointer for a while, passing it around, until eventually something else somewhere distant in the…
I think your first paragraph is too pessimistic. You can get a decent idea of the behaviour of the programs you use when they run out of memory by running under 'ulimit -v' with a low limit. In my experience most (though far from all) of the programs I use managed a clean abort with an error message (as from a traditional xmalloc()).
Most of these were likely managed-language programs.
Programs witten for managed language runtimes will have a language-runtime-level abort on malloc(2) fail, which usually is well written, in the sense that it will clean up language-runtime-level resources, and emit a language-runtime-level error message.
But this language-runtime-level abort usually isn’t exposed to the application in any hookable way, so from the developer’s perspective, it’s basically the same as being OOM killed. There’s no option to clean up e.g. an individual transaction’s resources in order to keep going. There’s no hooks for libraries to use to e.g. properly send close messages on sockets (if the language runtime doesn’t do that itself as part of managing socket lifetimes.) Etc.
These managed runtimes (e.g. the JVM) may expose a catchable exception for OOM errors, but these are for internal, language-runtime level OOM errors, triggered by the runtime itself under certain conditions, rather than in response to a syscall failure. When malloc(2) fails, it’s basically “too late” from these runtimes’ perspectives — they no longer have the resources required to allow the user to run any more code.
Re: PostgreSQL, Memory and the Cloud
#10Are there recommendations for learning about Linux kernel memory management? Two anecdata:
* I had some compute servers that were up for 200 days. The customers noticed that they were half as fast as identical hardware just booted. Dropping the file system cache ("echo 3 | sudo dd of=/proc/sys/vm/drop_cache") brought the speed back up to the newly deployed servers. WTF? File system caches are supposed to be zero cost discards as soon as processes ask for RAM - but something else is going on. I suspect the kernel is behaving badly with overpopulated RAM management data (TLB entries?), but I don't know how to measure that.
* If that is actually the problem, then a solution might be to decrease data size by using non-zero hugepages ("cat /proc/sys/vm/nr_hugepages"). I'd love to see recommendations on when to use that.