51–60 of 78 posts
Earlier quoted context omitted.
It does? Last I checked Linux doesn't do dynamic swap sizes, and while Windows has dynamic swap sizes it has a separate big non-dynamic file for hibernation. I have no idea what MacOS does.
You guys are getting Linux to hibernate!?
Earlier quoted context omitted.
Windows doesn't do that, though. If a process starts thrashing the performance goes to shit, but you can still operate the machine to kill it manually. Linux though? Utterly impossible. Usually even the desktop environment dies and I'm left with a blinking cursor. What good is it to get marginally better performance under low memory pressure at the cost of having to reboot the machine under extremely high memory pres…
In my experience the situations where you run into thrashing are rather rare nowadays. I personally wouldn't give up a good optimization for the rare worst case. (There's probably some knobs to turn as well, but I haven't had the need to figure that out.)
I think more people should know about the existence of ZRAM on modern Linux distributions. It's really changed the way I look at swap configs. ZRAM is a compressed block device that is stored in RAM. It's great! Previously, if I ever had high memory pressure situations, I really dreaded the slowdowns. Now, with swap sitting on top of /dev/zram0 it's a completely different experience. I have ZRAM enabled on all of my…
(It's fun to note that I try to type out "virtual memory" in this thread, because I don't want people to think I talk about virtual machines.)
Early BSD VM pre-allocated swap backing for every anonymous page — you couldn't allocate virtual memory without a swap slot reserved for it, even if the page was never paged out. When a process forks, the child needed swap reservations for the parent's entire address space (before exec replaces it). A large process forking temporarily needs double its swap allocation. If your working set is roughly equal to physical…
TBF, I think overcommit was and remains an ugliness in how we manage memory. I wish we'd solved the fork commit-charge-spike issue by encouraging vfork (and later, posix_spawn) more heavily, not by making the OS lie about the availability of memory. The ship's long sailed though, so even I run with overcommit enabled and only grumble about what might have been.
Honestly, I think overcommit is a good thing. If you want to give a process an isolated address space, then you have to allow that process to lay out memory as it sees fit, without having to worry too much about what else happens to be on the system. If you immediately "charge" the process for this, you will end up nit-picking every process on the system, even though with overcommit you would have been fine.
Earlier quoted context omitted.
In my experience the situations where you run into thrashing are rather rare nowadays. I personally wouldn't give up a good optimization for the rare worst case. (There's probably some knobs to turn as well, but I haven't had the need to figure that out.)
Try doing cargo build on a large Rust codebase with a matching number of CPU cores and GBs of RAM.
Earlier quoted context omitted.
There's a chance that those servers might run more efficiently with some swap space, for the reasons mentioned many times in this thread. Swap space is not just for overcommitting.
The theories are repeated a often but I have never seen any empirical data to back it up assuming one is setting the options I mentioned. These anecdotes usually come from servers with default settings and no attempt to tune them for the intended workloads and no capacity planning for application resources. Even OS maintainers are starting to recognize this and have created daemons such as tuned for the people that n…
Earlier quoted context omitted.
The theories are repeated a often but I have never seen any empirical data to back it up assuming one is setting the options I mentioned. These anecdotes usually come from servers with default settings and no attempt to tune them for the intended workloads and no capacity planning for application resources. Even OS maintainers are starting to recognize this and have created daemons such as tuned for the people that n…
Oh sure, it might or might not make a significant difference at all. Chances are, if you do a lot of I/O on a large (or very large) amount of data, and you also have a lot of rarely used but resident anonymous memory, then swap space should help, as that anonymous memory can get paged out in favor of disk cache, but I have no idea how common that is.
Earlier quoted context omitted.
Try doing cargo build on a large Rust codebase with a matching number of CPU cores and GBs of RAM.
I believe that it's not very hard to intentionally get into that situation, but... if you notice it doesn't work, won't you just not ? (It's not that this will work without swap after all, just OOM-kill without thrashing-pain.)
Earlier quoted context omitted.
Oh sure, it might or might not make a significant difference at all. Chances are, if you do a lot of I/O on a large (or very large) amount of data, and you also have a lot of rarely used but resident anonymous memory, then swap space should help, as that anonymous memory can get paged out in favor of disk cache, but I have no idea how common that is.
Yeah I mean, I know what you mean but this is where it gets into circular reasoning. I will always have operations groups move the workload to a node that has more memory if that is what is needed. In my case having swap on disk would require it to be encrypted due to contracts requiring any customer data touching a disk to be encrypted but I just avoid that all together and just add more memory. If 2TB or RAM isn't…
Earlier quoted context omitted.
I believe that it's not very hard to intentionally get into that situation, but... if you notice it doesn't work, won't you just not ? (It's not that this will work without swap after all, just OOM-kill without thrashing-pain.)
I don't intentionally configure crash-prone VMs. I have multiple concerns to juggle and can't always predict with certainty the best memory configuration. My point is that Linux should be able to deal with this situation without shitting the bed. It sucks to have some unsaved work in one window while another has decided that now would be a good time to turn the computer unusable. Like I said before, trading instabili…