121–130 of 140 posts
Earlier quoted context omitted.
Because no one can predict the future, and they don't know how many resources they will need.
I mean they can just malloc when they need more, what am I missing? Unless this is about JVMs which might preallocate a ton for their heaps and not use it
But it’s about 1000x slower to call the OS to allocate a new page, than to just use a pointer to preallocated space.
Earlier quoted context omitted.
Yeah, crash ballast is an extremely underrated tool. It also works for the OOM killer: run a daemon with a child process that holds some fixed amount of memory. Adjust OOM scores of everything else on the system lower than the child. If the parent’s waitpid() returns due to an OOM kill, send an alert/shutdown nonessentials/sync buffers to disk and so on.
Maybe you should skip running a useless child process and just use PSI to monitor memory pressure.
Earlier quoted context omitted.
Maybe you should skip running a useless child process and just use PSI to monitor memory pressure.
PSI polling is also good, but there’s no substitute for knowing for sure that the OOM killer is awake and hunting.
Earlier quoted context omitted.
PSI polling is also good, but there’s no substitute for knowing for sure that the OOM killer is awake and hunting.
You can't know when the OOM killer is awake and hunting. The kernel doesn't know that.
Earlier quoted context omitted.
> Both Windows and macOS do so much better out of the box for essentially any workload. We test FreeBSD, Linux, macOS, NetBSD, OpenBSD, and Windows in Zig's CI fleet. Of these, Windows is the only OS that we've had to configure with swap double the size of physical RAM to not hit completely unjustifiable OOMs. By "unjustifiable", I mean that we're not even close to actually running out of physical memory (let alone s…
When will people learn? NT is optimized, since day one, to swap. This is a feature, not a bug.
I'm not sure what point you're making here. Did you assume we had the page file disabled or something?
> This is a feature, not a bug.
OOMing when there's tens of gigabytes of unused memory is a feature...?
Firstly, if you take what's written at face value, it seems there's a serious logic error in the OOM killer. If it genuinely counts shared memory against a process, then its logic is wrong because killing that process wouldn't release that much memory, it'd need to kill all the processes sharing that memory to release it. So, maybe it should ignore shared memory in its calculations, or weight them by number of processes sharing it, or whatever.
The other issue is kind of true, but shows a stubbornness from the developers to change how they approach the problem. It's true that if any task could be partially through updating shared memory when it is killed, then all bets are off as to the state of that memory. However, if that's the case, then there should already be some kind of locking mechanisms in place to prevent multiple processes updating the same pages anyway.
Probably the current solution is: something gets locked when modified; every other process would need to spinlock until it's released; locking process is killed; everything else is stuck; another PG thread notices the child has died and kills everything else; on restart the DB has to be recovered.
A different solution could be: give every process its own private part of the shared memory for when it starts a transaction; have an indirection table from page number to shared memory page; for every page that needs to be modified, a new page is allocated from the freed pages list; that allocation is recorded in the private part of the shared memory along with the page number it's replacing; the old page is left untouched and copied to the new page along with any changes; the change list is terminated; then as the last step we update the indirection table for every page that was modified. If the process was killed at any point, we can either roll back the entirety of the transaction (freeing every allocation it made for replacement pages), or if the list was marked as terminated, finish off updating the indirection table with the changes required and marking the original pages as unused. At that point, the lock can be released knowing that shared memory is still entirely consistent.
Some of that is probably happening anyway if postgres supports reading from tables concurrently with an active write transaction on the same table, in which case the logic on how to mark those now freed pages as still in use is required. In that case, each process can also maintain a list of pages it has marked as still being used in case a reading process is killed off.
I have a couple of points, not really sure if they should be in one post or not, but whatever. Firstly, if you take what's written at face value, it seems there's a serious logic error in the OOM killer. If it genuinely counts shared memory against a process, then its logic is wrong because killing that process wouldn't release that much memory, it'd need to kill all the processes sharing that memory to release it. S…
I'd be interested to see a Linux distribution whose entire shtick is to run well-behaved under a kernel with overcommit disabled. But it would be a huge undertaking. Besides the obvious issue with fork(), there are a lot of programs and libraries out there that implicitly rely on overcommit due to not checking malloc() for failure.
There is some kind of illusion or myth that strict overcommit solves memory management issues.
Earlier quoted context omitted.
There is some kind of illusion or myth that strict overcommit solves memory management issues.
Not before userspace software is fixed, it doesn't.