I can suggest measuring working set size (WSS) instead of RSS.
See https://docs.kernel.org/admin-guide/mm/multigen_lru.html and https://docs.kernel.org/mm/damon/index.html
Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
11–20 of 28 posts
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#12Have you measured Pressure Stall Information or active pages from /proc/meminfo? Attempting to enumerate every resource variable (CPU, IOPS, RSS, Disk, logical count) into a single scoring function feels like an NP-hard trap. That's perfect for machine learning.
I admit PSI wasn't on our radar for this specific issue. We've been staring at RSS and page fault counters, but they are indeed too noisy in an mmap-heavy workload.
Checking /proc/pressure/memory to distinguish between 'healthy caching' and 'thrashing' sounds exactly like the signal we are missing. We will try to incorporate some pressure metrics into the node's health report. Thanks for the pointer.
But still, too many metrics for us to balance
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#13There's a simple solution: don't use mmap(). There's a reason that databases use O_DIRECT to read into their own in memory cache. If it was Good Enough for Oracle in the 1990s, it's probably Good Enough for you. mmap() is one of those things that looks like it's an easy solution when you start writing an application, but that's only because you don't know the complexity time bomb of what you're undertaking. The entir…
You're right. O_DIRECT is the endgame, but that's a full engine rewrite for us.
We're trying to stabilize the current architecture first. The complexity of hidden page fault blocking is definitely what's killing us, but we have to live with mmap for now.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#14> Coordinator sees Node A has significantly fewer rows (logical count) than the cluster average. It flags Node A as "underutilized." Ok, so you are dealing with a classic - you measure A, but what matters is B. For "load" balancing a decent metric is, well, response time (and jitter). For data partitioning - I guess number of rows is not the right metric? Change it to number*avg_size or something? If you can't measur…
The trouble with mmap is the performance cliff. A node goes from 'fine' to 'dead' almost instantly, which breaks our balancing logic.
You are right that we need better backpressure. Instead of a smarter coordinator, we probably need 'dumber' nodes that aggressively shed load (return 429s) the moment local pressure spikes, rather than waiting for a re-balance.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#15Have you measured Pressure Stall Information or active pages from /proc/meminfo? Attempting to enumerate every resource variable (CPU, IOPS, RSS, Disk, logical count) into a single scoring function feels like an NP-hard trap. That's perfect for machine learning.
Don't use Active/Inactive pages from /proc/meminfo. They don't represent the actual size of active/inactive memory.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#16There's a simple solution: don't use mmap(). There's a reason that databases use O_DIRECT to read into their own in memory cache. If it was Good Enough for Oracle in the 1990s, it's probably Good Enough for you. mmap() is one of those things that looks like it's an easy solution when you start writing an application, but that's only because you don't know the complexity time bomb of what you're undertaking. The entir…
You're right. O_DIRECT is the endgame, but that's a full engine rewrite for us. We're trying to stabilize the current architecture first. The complexity of hidden page fault blocking is definitely what's killing us, but we have to live with mmap for now.
I am curious -- what is the application and the language it's written in?
There are insanely dirty hacks that you could do to start controlling the fallout of the page faults (like playing games with userfaultfd), but they're unmaintainable in the long term as they introduce a fragility that results in unexpected complexity at the worst possible times (bugs). Rewriting / refactoring is not that hard once one understands the pattern, and I've done that quite a few times. Depending on the language, there may be other options. Doing an mlock() on the memory being used could help, but then it's absolutely necessary to carefully limit how much memory is pinned by such mappings.
Having been a kernel developer for a long time makes it a lot easier to spot what will work well for applications versus what can be considered glass jaws.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#17This is where having a little bit of swap can help you out. Not because you need swap, but because swap use % and swap I/O rates are good indicators. Something like 512 MB to maybe 1 G; not something like 2x your memory (unless you're on a very small system, and then use min(2x memory, 512 MB); having too much swap extends the amount of time your system can be swapping to death before it actually dies.
If your swap use jumps 10 points in a small time frame, you are running out of memory quickly. If your swap use hits 50 % or 80% or [whatever threshold], without any big jumps you're running out of memory slowly.
If your swap I/O is all output, not a huge deal... you're swapping stuff you never read. If you've got a lot of swapping in, chances are you're swapping to death.
> The Core Problem: We are trying to write a "God Equation" for our load balancer. We started with row_count, which failed. We looked at disk usage, but that doesn't correlate with RAM because of lazy loading.
I'm a big fan of straight up even distribution of requests. It's simple and predictable, although it's not going to get you the best throughput, predictability and simplicity is often better than perfection. If you always send each node 1/Nth of requests, worst case of a node that is broken but looks up is that you're still sending it a share when it should get nothing; if you have some sort of utilization based metric, if it looks underutilized because it's just dropping requests and responding with success but empty, it sucks up all your requests. Alternatively, people have good results with select M nodes by metrics, and then random selection between those. But also, IMHO, you want to reduce the work your load balancer(s) do, because load balancing load balancers is hard.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#18There's a simple solution: don't use mmap(). There's a reason that databases use O_DIRECT to read into their own in memory cache. If it was Good Enough for Oracle in the 1990s, it's probably Good Enough for you. mmap() is one of those things that looks like it's an easy solution when you start writing an application, but that's only because you don't know the complexity time bomb of what you're undertaking. The entir…
There is a database that uses `mmap()` - RavenDB. Their memory accounting is utter horror - they somehow use Commited_AS from /proc/meminfo in their calculations. Their recommendation to avoid OOMs is to have swap twice the size of RAM. Their Jepsen test results are pure comedy.
LMDB uses mmap() as well, but it only supports one process holding the database open at a time. It's also not intended for working sets larger than available RAM.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#19Earlier quoted context omitted.
There is a database that uses `mmap()` - RavenDB. Their memory accounting is utter horror - they somehow use Commited_AS from /proc/meminfo in their calculations. Their recommendation to avoid OOMs is to have swap twice the size of RAM. Their Jepsen test results are pure comedy.
LMDB uses mmap() as well, but it only supports one process holding the database open at a time. It's also not intended for working sets larger than available RAM.
Wrong, LMDB fully supports multiprocess concurrency as well as DBs multiple orders of magnitude larger than RAM. Wherever you got your info from is dead wrong.
Among embedded key/value stores, only LMDB and BerkeleyDB support multiprocess access. RocksDB, LevelDB, etc. are all single process.
Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie
#20Earlier quoted context omitted.
LMDB uses mmap() as well, but it only supports one process holding the database open at a time. It's also not intended for working sets larger than available RAM.
Wrong, LMDB fully supports multiprocess concurrency as well as DBs multiple orders of magnitude larger than RAM. Wherever you got your info from is dead wrong. Among embedded key/value stores, only LMDB and BerkeleyDB support multiprocess access. RocksDB, LevelDB, etc. are all single process.
My mistake. Doesn’t it have a global lock though?
Also, even if LMDB supports databases larger than RAM, that’s it doesn’t mean it’s a good idea to have a working set that exceeds that size. Unless you’re claiming it’s scan resistant?