21–30 of 60 posts
Earlier quoted context omitted.
Are you using any swap? If so, check the swappiness setting
No swap. These are large RAM (400G to 1000G) Kubernetes nodes.
Wow, the title of this post is very calm compared to what is actually happening. CloudSQL Postgres is running with a misconfigured OS OOM killer, crashes Postmaster randomly even if memory use is below instance spec. GCP closes this bug report as "Won't fix". This is a priority 1 issue. Seeing a wontfix for this has completely destroyed my trust of their judgement. The bug report states that they have been in contact…
It exposes a very problematic communication pattern. The engineering team doesn't respond to the support team (accidentally or deliberately). The support team then just decides to close the issue instead of prodding the engineering team for an actual response (even if it's just "Yeah, we're not fixing it"). Now the issue is just in limbo and the only one who feels the pain is the customer.
I've observed with with Atlassian where I wanted to report a Jira bug, but found that it had already been opened some years before, more than a hundred people had subscribed, bug was still closed as "no activity, must not be relevant". I just found the exact same bug reported for Jira Cloud (I had observed it in the on-prem version): https://jira.atlassian.com/browse/JSWCLOUD-8865 and it was closed there for the very same reason.
I didn't leave a comment because the original report described the issue perfectly, and adding a "me too" comment is just noise in the bug tracker. Guess I'll be noise in future :-(
So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
Some programs allocate a lot of virtual memory and then don't use it. Also, linux's forking model can result in a lot of virtual memory being allocated if a heavy-weight program tries to fork+exec a lot of smaller programs, since fork+exec it not atomic and briefly doubles the virtual memory usage of the original program. I think there are better ways to spawn programs that don't suffer from this problem now... If yo…
So are there problems with disabling overcommit? Or is it really that simple (at least for dedicated db hosts)?
Consider this scenario: a process runs a fork(), and shortly after it runs an exec(). Normally, the extra fork only uses a tiny amount of extra memory, because the memory is shared between the parent and the child, until one of them writes to it (copy-on-write).
With overcommit disabled, the kernel must reserve enough space to copy the whole writable RAM of a process when it forks.
So you have a 16GB machine, and an 8.1GB process cannot spawn any other program through the usual fork + exec routine (workarounds exist, like forking before allocating lots of memory and using IPC to instruct the low-memory fork to fork again and launch, but that's way more complicated than a simple fork + exec).
So if you have a dedicated DB host and you know that your DB engine is very carefully engineered to work with disabled overcommit, you can disable it. On a general-purpose machine a disabled overcommit will waste lots of RAM that's sitting unused.
Are there recommendations for learning about Linux kernel memory management? Two anecdata: * I had some compute servers that were up for 200 days. The customers noticed that they were half as fast as identical hardware just booted. Dropping the file system cache ("echo 3 | sudo dd of=/proc/sys/vm/drop_cache") brought the speed back up to the newly deployed servers. WTF? File system caches are supposed to be zero cost…
I don’t remember details now, but I’ve seen a situation when a Java app was working slower and a box with more RAM (and probably a bigger heap size), compare to a box with the same CPU but 2x less RAM. I suspected that TLB cache was the reason, but didn’t have time to test this.
Earlier quoted context omitted.
It exposes a very problematic communication pattern. The engineering team doesn't respond to the support team (accidentally or deliberately). The support team then just decides to close the issue instead of prodding the engineering team for an actual response (even if it's just "Yeah, we're not fixing it"). Now the issue is just in limbo and the only one who feels the pain is the customer.
Another "fun" interaction pattern: User reports a bug (or a feature request), several others subscribe to and/or vote for this to be solved, and then a service rep closes the issue because there wasn't any recent activity. I've observed with with Atlassian where I wanted to report a Jira bug, but found that it had already been opened some years before, more than a hundred people had subscribed, bug was still closed a…
Are there recommendations for learning about Linux kernel memory management? Two anecdata: * I had some compute servers that were up for 200 days. The customers noticed that they were half as fast as identical hardware just booted. Dropping the file system cache ("echo 3 | sudo dd of=/proc/sys/vm/drop_cache") brought the speed back up to the newly deployed servers. WTF? File system caches are supposed to be zero cost…
For other processes you'll need a hugepage-aware allocator such as tcmalloc (the new one, not the old one) and transparent hugepages enabled. Again, the benefits of this may be enormous, if page table management is expensive on your services.
You will find a great deal of blogs on the web recommending disabling transparent hugepages. These people are all mislead. Hugepages are a major benefit.
Earlier quoted context omitted.
Some programs allocate a lot of virtual memory and then don't use it. Also, linux's forking model can result in a lot of virtual memory being allocated if a heavy-weight program tries to fork+exec a lot of smaller programs, since fork+exec it not atomic and briefly doubles the virtual memory usage of the original program. I think there are better ways to spawn programs that don't suffer from this problem now... If yo…
Wasn't vfork designed to solve this issue? Does it not work in practice?
Are there recommendations for learning about Linux kernel memory management? Two anecdata: * I had some compute servers that were up for 200 days. The customers noticed that they were half as fast as identical hardware just booted. Dropping the file system cache ("echo 3 | sudo dd of=/proc/sys/vm/drop_cache") brought the speed back up to the newly deployed servers. WTF? File system caches are supposed to be zero cost…